Author name cluster

Jana Kosecka

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

23 papers

2 author rows

ICRA Conference 2025 Conference Paper

A Hybrid Approach to Indoor Social Navigation: Integrating Reactive Local Planning and Proactive Global Planning

Arnab Debnath
Gregory J. Stein
Jana Kosecka

We consider the problem of indoor building-scale social navigation, where the robot must reach a point goal as quickly as possible without colliding with humans who are freely moving around. Factors such as varying crowd densities, unpredictable human behavior, and the constraints of indoor spaces add significant complexity to the navigation task, necessitating a more advanced approach. We propose a modular navigation framework that leverages the strengths of both classical methods and deep reinforcement learning (DRL). Our approach employs a global planner to generate waypoints, assigning soft costs around anticipated pedestrian locations, encouraging caution around potential future positions of humans. Simultaneously, the local planner, powered by DRL, follows these waypoints while avoiding collisions. The combination of these planners enables the agent to perform complex maneuvers and effectively navigate crowded and constrained environments while improving reliability. Many existing studies on social navigation are conducted in simplistic or open environments, limiting the ability of trained models to perform well in complex, real-world settings. To advance research in this area, we introduce a new 2D benchmark designed to facilitate development and testing of social navigation strategies in indoor environments. 2 2 Simulator and code: https://github.com/arnabGMU/hybrid_social_nav We benchmark our method against traditional and RL-based navigation strategies, demonstrating that our approach outperforms both.

IROS Conference 2023 Conference Paper

Learning-Augmented Model-Based Planning for Visual Exploration

Yimeng Li
Arnab Debnath
Gregory J. Stein
Jana Kosecka

We consider the problem of time-limited robotic exploration in previously unseen environments where exploration is limited by a predefined amount of time. We propose a novel exploration approach using learning-augmented model-based planning. We generate a set of sub goals associated with frontiers on the current map and derive a Bellman Equation for exploration with these subgoals. Visual sensing and advances in semantic mapping of indoor scenes are exploited for training a deep convolutional neural network to estimate properties associated with each frontier: the expected unobserved area beyond the frontier and the expected time steps (discretized actions) required to explore it. The proposed model-based planner is guaranteed to explore the whole scene if time permits. We thoroughly evaluate our approach on a large-scale pseudo-realistic indoor dataset (Matterport3D) with the Habitat simulator. We compare our approach with classical and more recent RL-based exploration methods. Our approach surpasses the greedy strategies by 2. 1% and the RL-based exploration methods by 8. 4% in terms of coverage.

IROS Conference 2022 Conference Paper

Object Pose Estimation using Mid-level Visual Representations

Negar Nejatishahidin
Pooya Fayyazsanavi
Jana Kosecka

This work proposes a novel pose estimation model for object categories that can be effectively transferred to pre-viously unseen environments. The deep convolutional network models (CNN) for pose estimation are typically trained and evaluated on datasets specifically curated for object detection, pose estimation, or 3D reconstruction, which requires large amounts of training data. In this work, we propose a model for pose estimation that can be trained with small amount of data and is built on the top of generic mid-level represen-tations [33] (e. g. surface normal estimation and re-shading). These representations are trained on a large dataset without requiring pose and object annotations. Later on, the predictions are refined with a small CNN neural network that exploits object masks and silhouette retrieval. The presented approach achieves superior performance on the Pix3D dataset [26] and shows nearly 35 % improvement over the existing models when only 25 % of the training data is available. We show that the approach is favorable when it comes to generalization and transfer to novel environments. Towards this end, we introduce a new pose estimation benchmark for commonly encountered furniture categories on challenging Active Vision Dataset [1] and evaluated the models trained on the Pix3D dataset.

AAAI Conference 2020 Short Paper

American Sign Language Recognition Using an FMCW Wireless Sensor (Student Abstract)

Yuanqi Du
Nguyen Dang
Riley Wilkerson
Parth Pathak
Huzefa Rangwala
Jana Kosecka

In today’s digital world, rapid technological advancements continue to lessen the burden of tasks for individuals. Among these tasks is communication across perceived language barriers. Indeed, increased attention has been drawn to American Sign Language (ASL) recognition in recent years. Camerabased and motion detection-based methods have been researched extensively; however, there remains a divide in communication between ASL users and non-users. Therefore, this research team proposes the use of a novel wireless sensor (Frequency-Modulated Continuous-Wave Radar) to help bridge the gap in communication. In short, this device sends out signals that detect the user’s body positioning in space. These signals then reﬂect off the body and back to the sensor, developing thousands of cloud points per second, indicating where the body is positioned in space. These cloud points can then be examined for movement over multiple consecutive time frames using a cell division algorithm, ultimately showing how the body moves through space as it completes a single gesture or sentence. At the end of the project, 95% accuracy was achieved in one-object prediction as well as 80% accuracy on cross-object prediction with 30% other objects’ data introduced on 19 commonly used gestures. There are 30 samples for each gesture per person from three persons.

ICRA Conference 2020 Conference Paper

Learning View and Target Invariant Visual Servoing for Navigation

Yimeng Li
Jana Kosecka

The advances in deep reinforcement learning recently revived interest in data-driven learning based approaches to navigation. In this paper we propose to learn viewpoint invariant and target invariant visual servoing for local mobile robot navigation; given an initial view and the goal view or an image of a target, we train deep convolutional network controller to reach the desired goal. We present a new architecture for this task which rests on the ability of establishing correspondences between the initial and goal view and novel reward structure motivated by the traditional feedback control error. The advantage of the proposed model is that it does not require calibration and depth information and achieves robust visual servoing in a variety of environments and targets without any parameter fine tuning. We present comprehensive evaluation of the approach and comparison with other deep learning architectures as well as classical visual servoing methods in visually realistic simulation environment [1]. The presented model overcomes the brittleness of classical visual servoing based methods and achieves significantly higher generalization capability compared to the previous learning approaches.

ICRA Conference 2019 Conference Paper

Visual Representations for Semantic Target Driven Navigation

Arsalan Mousavian
Alexander Toshev
Marek Fiser
Jana Kosecka
Ayzaan Wahid
James Davidson

What is a good visual representation for navigation? We study this question in the context of semantic visual navigation, which is the problem of a robot finding its way through a previously unseen environment to a target object, e. g. go to the refrigerator. Instead of acquiring a metric semantic map of an environment and using planning for navigation, our approach learns navigation policies on top of representations that capture spatial layout and semantic contextual cues. We propose to use semantic segmentation and detection masks as observations obtained by state-of-the-art computer vision algorithms and use a deep network to learn the navigation policy. The availability of equitable representations in simulated environments enables joint training using real and simulated data and alleviates the need for domain adaptation or domain randomization commonly used to tackle the sim-to-real transfer of the learned policies. Both the representation and the navigation policy can be readily applied to real non-synthetic environments as demonstrated on the Active Vision Dataset [1]. Our approach successfully gets to the target in 54% of the cases in unexplored environments, compared to 46% for a non-learning based approach, and 28% for a learning-based baseline.

IROS Conference 2018 Conference Paper

FarSight: Long-Range Depth Estimation from Outdoor Images

Md. Alimoor Reza
Jana Kosecka
Philip David

This paper introduces the problem of long-range monocular depth estimation for outdoor urban environments. Range sensors and traditional depth estimation algorithms (both stereo and single view) predict depth for distances of less than 100 meters in outdoor settings and 10 meters in indoor settings. The shortcomings of outdoor single view methods that use learning approaches are, to some extent, due to the lack of long-range ground truth training data, which in turn is due to limitations of range sensors. To circumvent this, we first propose a novel strategy for generating synthetic long-range ground truth depth data. We utilize Google Earth images to reconstruct large-scale 3D models of different cities with proper scale. The acquired repository of 3D models and associated RGB views along with their long-range depth renderings are used as training data for depth prediction. We then train two deep neural network models for long-range depth estimation: i) a Convolutional Neural Network (CNN) and ii) a Generative Adversarial Network (GAN). We found in our experiments that the GAN model predicts depth more accurately. We plan to open-source the database and the baseline models for public use.

ICRA Conference 2017 Conference Paper

A dataset for developing and benchmarking active vision

Phil Ammirato
Patrick Poirson
Eunbyung Park
Jana Kosecka
Alexander C. Berg

We present a new public dataset with a focus on simulating robotic vision tasks in everyday indoor environments using real imagery. The dataset includes 20, 000+ RGB-D images and 50, 000+ 2D bounding boxes of object instances densely captured in 9 unique scenes. We train a fast object category detector for instance detection on our data. Using the dataset we show that, although increasingly accurate and fast, the state of the art for object detection is still severely impacted by object scale, occlusion, and viewing direction all of which matter for robotics applications. We next validate the dataset for simulating active vision, and use the dataset to develop and evaluate a deep-network-based system for next best move prediction for object classification using reinforcement learning. Our dataset is available for download at cs. unc.edu/~ammirato/active_vision_dataset_website/.

IROS Conference 2017 Conference Paper

Dense piecewise planar RGB-D SLAM for indoor environments

Phi-Hung Le
Jana Kosecka

The paper exploits weak Manhattan constraints to parse the structure of indoor environments from RGB-D video sequences in an online setting. We extend the previous approach for single view parsing of indoor scenes to video sequences and formulate the problem of recovering the floor plan of the environment as an optimal labeling problem solved using dynamic programming. The temporal continuity is enforced in a recursive setting, where labeling from previous frames is used as a prior term in the objective function. In addition to recovery of piecewise planar weak Manhattan structure of the extended environment, the orthogonality constraints are also exploited by visual odometry and pose graph optimization. This yields reliable estimates in the presence of large motions and absence of distinctive features to track. We evaluate our method on several challenging indoors sequences demonstrating accurate SLAM and dense mapping of low texture environments. On existing TUM benchmark [19] we achieve competitive results with the alternative approaches which fail in our environments.

IROS Conference 2017 Conference Paper

Label propagation in RGB-D video

Md. Alimoor Reza
Hui Zheng
Georgios Georgakis
Jana Kosecka

We propose a new method for the propagation of semantic labels in RGB-D video of indoor scenes given a set of ground truth keyframes. Manual labeling of all pixels in every frame of a video sequence is labor intensive and costly, yet required for training and testing of semantic segmentation methods. The availability of video enables propagation of labels between the frames for obtaining a large amounts of annotated pixels. While previous methods commonly used optical flow motion cues for label propagation, we present a novel approach using the camera poses and 3D point clouds for propagating the labels in superpixels computed on the unannotated frames of the sequence. The propagation task is formulated as an energy minimization problem in a Conditional Random Field (CRF). We performed experiments on 8 video sequences from SUN3D dataset [1] and showed superior performance to an optical flow based label propagation approach. Furthermore, we demonstrated that the propagated labels can be used to learn better models using data hungry deep convolutional neural network (DCNN) based approaches for the task of semantic segmentation. The approach demonstrates an increase in performance when the ground truth keyframes are combined with the propagated labels during training.

IROS Conference 2016 Conference Paper

RGB-D multi-view object detection with object proposals and shape context

Georgios Georgakis
Md. Alimoor Reza
Jana Kosecka

We propose a novel approach for multi-view object detection in 3D scenes reconstructed from RGB-D sensor. We utilize shape based representation using local shape context descriptors along with the voting strategy which is supported by unsupervised object proposals generated from 3D point cloud data. Our algorithm starts with a single-view object detection where object proposals generated in 3D space are combined with object specific hypotheses generated by the voting strategy. To tackle the multi-view setting, the data association between multiple views enabled view registration and 3D object proposals. The evidence from multiple views is combined in simple bayesian setting. The approach is evaluated on the Washington RGB-D scenes datasets [1], [2] containing several classes of objects in a table top setting. We evaluated our approach against the other state-of-the-art methods and demonstrated superior performance on the same dataset.

ICRA Conference 2015 Conference Paper

Semantically guided location recognition for outdoors scenes

Arsalan Mousavian
Jana Kosecka
Jyh-Ming Lien

The problem of image based localization has a long history both in robotics and computer vision and shares many similarities with image based retrieval problem. Existing techniques use either local features or (semi)-global image signatures in the context of topological mapping or loop closure detection. Difficulties of the location recognition problem are often affected by large appearance and viewpoint variation between the query view and reference dataset and presence of non-discriminative features due to vegetation, sky and road. In this work we show that semantic segmentation labeling of man-made structures can inform the traditional bag-of-visual words models to obtain proper feature weighting and improve the overall location recognition accuracy. We also demonstrate additional capability of identifying individual buildings and estimating their extent in images, providing the essential building block for semantic localization. Towards this end we introduce a new challenging outdoors urban dataset exhibiting large variations in appearance and viewpoint.

ICRA Conference 2014 Conference Paper

Semantic segmentation with heterogeneous sensor coverages

Cesar Cadena 0001
Jana Kosecka

We propose a new approach to semantic parsing, which can seamlessly integrate evidence from multiple sensors with overlapping but possibly different fields of view (FOV), account for missing data and predict semantic labels over the spatial union of sensors coverages. The existing approaches typically carry out semantic segmentation using only one modality, incorrectly interpolate measurements of other modalities or at best assign semantic labels only to the spatial intersection of coverages of different sensors. In this work we remedy these problems by proposing an effective and efficient strategy for inducing the graph structure of Conditional Random Field used for inference and a novel method for computing the sensor domain dependent potentials. We focus on RGB cameras and 3D data from lasers or depth sensors. The proposed approach achieves superior performance, compared to state of the art and obtains labels for the union of spatial coverages of both sensors, while effectively using appearance or 3D cues when they are available. The efficiency of the approach is amenable to realtime implementation. We quantitatively validate our proposal in two publicly available datasets from indoors and outdoors real environments. The obtained semantic understanding of the acquired sensory information can enable higher level tasks for autonomous mobile robots and facilitate semantic mapping of the environments.

ICRA Conference 2012 Conference Paper

Acquiring semantics induced topology in urban environments

Gautam Singh
Jana Kosecka

Methods for acquisition and maintenance of an environment model are central to a broad class of mobility and navigation problems. Towards this end, various metric, topological or hybrid models have been proposed. Due to recent advances in sensing and recognition, acquisition of semantic models of the environments have gained increased interest in the community. In this work, we will demonstrate a capability of using weak semantic models of the environment to induce different topological models, capturing the spatial semantics of the environment at different levels. In the first stage of the model acquisition, we propose to compute semantic layout of the street scenes imagery by recognizing and segmenting buildings, roads, sky, cars and trees. Given such semantic layout, we propose an informative feature characterizing the layout and train a classifier to recognize street intersections in challenging urban inner city scenes. We also show how the evidence of different semantic concepts can induce useful topological representation of the environment, which can aid navigation and localization tasks. To demonstrate the approach, we carry out experiments on a challenging dataset of omnidirectional inner city street views and report the performance of both semantic segmentation and intersection classification.

IROS Conference 2011 Conference Paper

Label propagation in videos indoors with an incremental non-parametric model update

J. Rituerto
Ana C. Murillo
Jana Kosecka

Semantic interpretation of the environment can significantly improve the capabilities of our autonomous robots. This work is focused on automatic semantic label propagation in video of indoor environments acquired by a mobile robot. Using a small number of training examples, we propose a new approach to recognize and label dominant background regions of interest, such as floor, wall and doors, and separate them from the remaining of foreground/object image categories. Our approach performs the labeling at the level of image superpixels. A simple non-parametric model is initialized from a few hand labeled examples in the first frame, and then it is propagated and updated along the sequence. We demonstrate the promising results obtained with our proposal in five different indoor sequences from different environments. The obtained semantic labeling can be used both for autonomous navigation and to provide better context for subsequent object detection.

ICRA Conference 2006 Conference Paper

Probabilistic Location Recognition using Reduced Feature Set

Fayin Li
Jana Kosecka

The localization capability is central to basic navigation tasks and motivates development of various visual navigation systems. In this paper we describe a two stage approach for localization in indoor environments. In the first stage, the environment is partitioned into several locations, each characterized by a set of scale-invariant keypoints and their associated descriptors. In the second stage the keypoints of the query view are integrated probabilistically yielding an estimate of most likely location. The novelty of our approach is in the selection of discriminative features, best suited for characterizing individual locations. We demonstrate that high location recognition rate is maintained with only 10% of the originally detected features, yielding a substantial speedup in recognition and capability of handling larger environments. The ambiguities due to the self-similarity and dynamic changes in the environment are resolved by exploiting spatial relationships between locations captured by hidden Markov model

ICRA Conference 2004 Conference Paper

Vision based Topological Markov Localization

Jana Kosecka
Fayin Li

In this paper we study the problem of acquiring a topological model of indoors environment by means of visual sensing and subsequent localization given the model. The resulting model consists of a set of locations and neighborhood relationships between them. Each location in the model is represented by a collection of representative views and their associated descriptors selected from a temporally sub-sampled video stream captured by a mobile robot during exploration. We compare the recognition performance using global image histograms as well as local scale-invariant features as image descriptors, demonstrate their strengths and weaknesses and show how to model the spatial relationships between individual locations by a Hidden Markov Model. The quality of the acquired model is tested in the localization stage by means of location recognition: given a new view or a sequence of views, the most likely location where that view came from is determined.

IROS Conference 2003 Conference Paper

Communication enhanced navigation strategies for teams of mobile agents

Justin Hayes
Martha McJunkin
Jana Kosecka

In multi-agent systems engaged in cooperative activities there is an apparent trade-off between the complexity of the individual agents, their sensing capabilities and communication required for accomplishment of particular tasks. One of the main computationally intensive components which affects the complexity of the overall system is the acquisition and maintenance of the environment model where the agents reside. In this paper, in the context of foraging and coordinated traversal task, we will examine control strategies that in the absence of the global model of the environment can substantially improve the performance of the team using additional sensing and communication capabilities. In one case the coordinated strategy is motivated by an ant trail following behavior while in another case the line of sight information is used to constrain the movement of individual agents guaranteeing shorter total traversal times.

ICRA Conference 2002 Conference Paper

Efficient Computation of Vanishing Points

Jana Kosecka
Wei Zhang 0018

Man-made environments possess a lot of regularities which simplify otherwise difficult pose estimation and visual reconstruction tasks. The constraints arising front parallel and orthogonal lines and planes can be efficiently exploited at various stages of vision processing pipeline. In this paper we propose an approach for estimation of vanishing points by exploiting the constraints of structured man-made environments, where the majority of lines is aligned with the principal orthogonal directions of the world coordinate frame. We combine efficient image processing techniques used in the line detection and initialization stage with simultaneous grouping and estimation of vanishing directions using expectation maximization (EM) algorithm. Since we assume an uncalibrated camera the estimated vanishing points can be used towards partial camera calibration and estimation of the relative orientation of the camera with respect to the scene. The presented approach is computationally efficient and has been verified extensively by experiments.

ICRA Conference 1998 Conference Paper

A Comparative Study of Vision-Based Lateral Control Strategies for Autonomous Highway Driving

Jana Kosecka
Robert Blasi
Camillo J. Taylor
Jitendra Malik

This paper will present the results of a comparative study of a set of vision-based control strategies that have been applied to the problem of steering an autonomous vehicle along a highway. The aim of this work has been to further our understanding of the characteristics of various control laws that could be applied to this problem with a view to making informed design decisions. The control strategies that we explored include a lead lag control law, a full-state linear controller and input-output linearizing control law. Each of these control strategies was implemented and tested on our experimental vehicle, a Honda Accord LX, both with and without a curvature feedforward component.

IROS Conference 1997 Conference Paper

Generation of conflict resolution manoeuvres for air traffic management

Jana Kosecka
Claire J. Tomlin
George J. Pappas
S. Shankar Sastry

We explore the use of distributed online motion planning algorithms for multiple mobile agents, in air traffic management systems (ATMS). The work is motivated by current trends in ATMS to move towards decentralized air traffic management, in which the aircraft operate in "free flight" mode instead of following prespecified "sky freeways". Conflict resolution strategies are an integral part of the free flight setting. The purpose of this paper is to obtain a set of manoeuvres to cover all possible conflict scenarios involving multiple agents. A distributed motion planning algorithm based on potential and vortex fields is used. While the algorithm is not always guaranteed to generate flyable trajectories, the obtained trajectories can serve as qualitative prototypes for coordination manoeuvres between multiple aircraft. The actual manoeuvres are generated by approximating these prototypes with trajectories made zip of straight lines and are further verified using hybrid verification techniques.

IROS Conference 1995 Conference Paper

Cooperative material handling by human and robotic agents: module development and system synthesis

Julie A. Adams
Ruzena Bajcsy
Jana Kosecka
Vijay Kumar 0001
Robert Mandelbaum
Max Mintz
Richard P. Paul
Curtis Wang

Presents a collaborative effort to design and implement a cooperative material handling system by a small team of human and robotic agents in an unstructured indoor environment. The authors' approach makes fundamental use of the human agents' expertise for aspects of task planning, task monitoring, and error recovery. The authors' system is neither fully autonomous nor fully teleoperated. It is designed to make effective use of the human's abilities within the present state of the art of autonomous systems. The authors' robotic agents refer to systems which are each equipped with at least one sensing modality and which possess some capability for self-orientation and/or mobility. The authors' robotic agents are not required to be homogeneous with respect to either capabilities or function. The authors' research stresses both paradigms and testbed experimentation. Theory issues include the requisite coordination principles and techniques which are fundamental to a cooperative multiagent system's basic functioning. The authors have constructed an experimental distributed multiagent-architecture testbed facility. The required modular components of this testbed are currently operational and have been tested individually. The authors' current research focuses on the agents' integration in a scenario for cooperative material handling.

ICRA Conference 1994 Conference Paper

Application of Discrete Events Systems for Modeling and Controlling Robotic Agents

Jana Kosecka
Luca Bogoni

In this paper we present a framework for modeling behaviors and tasks for heterogeneous robotic agents. For this purpose we have adopted a formalism from the discrete events systems (DES) theory. We distinguish two kinds of scenarios. In the first one, reactive behaviors of mobile agents directly connect observations with actions. The overall objective is to achieve controllability of the system which is composed from modular components operating in parallel. In the second one, observations are implicitly connected with actions and the objective is to design an observer for manipulatory tasks which would guarantee the task's observability. The use of the DES formalism allows one to describe complex interactions between different components in a systematic fashion and guarantee some control-theoretic properties. We demonstrate our approach by presenting examples of navigation, obstacle avoidance, piercing and picking. >