Arrow Research search

Author name cluster

Laurent Itti

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

36 papers
2 author rows

Possible papers

36

ICLR Conference 2025 Conference Paper

DreamDistribution: Learning Prompt Distribution for Diverse In-distribution Generation

  • Brian Nlong Zhao
  • Yuhang Xiao
  • Jiashu Xu
  • Xinyang Jiang
  • Yifan Yang 0004
  • Dongsheng Li 0002
  • Laurent Itti
  • Vibhav Vineet

The popularization of Text-to-Image (T2I) diffusion models enables the generation of high-quality images from text descriptions. However, generating diverse customized images with reference visual attributes remains challenging. This work focuses on personalizing T2I diffusion models at a more abstract concept or category level, adapting commonalities from a set of reference images while creating new instances with sufficient variations. We introduce a solution that allows a pretrained T2I diffusion model to learn a set of soft prompts, enabling the generation of novel images by sampling prompts from the learned distribution. These prompts offer text-guided editing capabilities and additional flexibility in controlling variation and mixing between multiple distributions. We also show the adaptability of the learned prompt distribution to other tasks, such as text-to-3D. Finally we demonstrate effectiveness of our approach through quantitative analysis including automatic evaluation and human assessment.

Details

ICRA Conference 2024 Conference Paper

Open X-Embodiment: Robotic Learning Datasets and RT-X Models: Open X-Embodiment Collaboration

  • Abby O'Neill
  • Abdul Rehman
  • Abhiram Maddukuri
  • Abhishek Gupta 0004
  • Abhishek Padalkar
  • Abraham Lee
  • Acorn Pooley
  • Agrim Gupta

Large, high-capacity models trained on diverse datasets have shown remarkable successes on efficiently tackling downstream applications. In domains from NLP to Computer Vision, this has led to a consolidation of pretrained models, with general pretrained backbones serving as a starting point for many applications. Can such a consolidation happen in robotics? Conventionally, robotic learning methods train a separate model for every application, every robot, and even every environment. Can we instead train "generalist" X-robot policy that can be adapted efficiently to new robots, tasks, and environments? In this paper, we provide datasets in standardized data formats and models to make it possible to explore this possibility in the context of robotic manipulation, alongside experimental results that provide an example of effective X-robot policies. We assemble a dataset from 22 different robots collected through a collaboration between 21 institutions, demonstrating 527 skills (160266 tasks). We show that a high-capacity model trained on this data, which we call RT-X, exhibits positive transfer and improves the capabilities of multiple robots by leveraging experience from other platforms. The project website is robotics-transformer-x. github.io.

Details

NeurIPS Conference 2024 Conference Paper

USCILab3D: A Large-scale, Long-term, Semantically Annotated Outdoor Dataset

  • Kiran Lekkala
  • Henghui Bao
  • Peixu Cai
  • Wei Z. Lim
  • Chen Liu
  • Laurent Itti

In this paper, we introduce the \textbf{USCILab3D dataset}, a large-scale, annotated outdoor dataset designed for versatile applications across multiple domains, including computer vision, robotics, and machine learning. The dataset was acquired using a mobile robot equipped with 5 cameras and a 32-beam, $360^{\circ}$ scanning LIDAR. The robot was teleoperated, over the course of a year and under a variety of weather and lighting conditions, through a rich variety of paths within the USC campus (229 acres = $\sim 92. 7$ hectares). The raw data was annotated using state-of-the-art large foundation models, and processed to provide multi-view imagery, 3D reconstructions, semantically-annotated images and point clouds (267 semantic categories), and text descriptions of images and objects within. The dataset also offers a diverse array of complex analyses using pose-stamping and trajectory data. In sum, the dataset offers 1. 4M point clouds and 10M images ($\sim 6$TB of data). Despite covering a narrower geographical scope compared to a whole-city dataset, our dataset prioritizes intricate intersections along with denser multi-view scene images and semantic point clouds, enabling more precise 3D labelling and facilitating a broader spectrum of 3D vision tasks. For data, code and more details, please visit our website.

PDF Details DOI

NeurIPS Conference 2023 Conference Paper

3D Copy-Paste: Physically Plausible Object Insertion for Monocular 3D Detection

  • Yunhao Ge
  • Hong-Xing Yu
  • Cheng Zhao
  • Yuliang Guo
  • Xinyu Huang
  • Liu Ren
  • Laurent Itti
  • Jiajun Wu

A major challenge in monocular 3D object detection is the limited diversity and quantity of objects in real datasets. While augmenting real scenes with virtual objects holds promise to improve both the diversity and quantity of the objects, it remains elusive due to the lack of an effective 3D object insertion method in complex real captured scenes. In this work, we study augmenting complex real indoor scenes with virtual objects for monocular 3D object detection. The main challenge is to automatically identify plausible physical properties for virtual assets (e. g. , locations, appearances, sizes, etc. ) in cluttered real scenes. To address this challenge, we propose a physically plausible indoor 3D object insertion approach to automatically copy virtual objects and paste them into real scenes. The resulting objects in scenes have 3D bounding boxes with plausible physical locations and appearances. In particular, our method first identifies physically feasible locations and poses for the inserted objects to prevent collisions with the existing room layout. Subsequently, it estimates spatially-varying illumination for the insertion location, enabling the immersive blending of the virtual objects into the original scene with plausible appearances and cast shadows. We show that our augmentation method significantly improves existing monocular 3D object models and achieves state-of-the-art performance. For the first time, we demonstrate that a physically plausible 3D object insertion, serving as a generative data augmentation technique, can lead to significant improvements for discriminative downstream tasks such as monocular 3D object detection. Project website: https: //gyhandy. github. io/3D-Copy-Paste/.

PDF Details

TMLR Journal 2023 Journal Article

Invariant Structure Learning for Better Generalization and Causal Explainability

  • Yunhao Ge
  • Sercan O Arik
  • Jinsung Yoon
  • Ao Xu
  • Laurent Itti
  • Tomas Pfister

Learning the causal structure behind data is invaluable for improving generalization and ob- taining high-quality explanations. Towards this end, we propose a novel framework, Invariant Structure Learning (ISL), that is designed to improve causal structure discovery by utilizing generalization as an indication in the process. ISL splits the data into different environments, and learns a structure that is invariant to the target across different environments by imposing a consistency constraint. The proposed aggregation mechanism then selects the classifier based on a graph structure that reflects the causal mechanisms in the data more accurately compared to the structures learnt from individual environments. Furthermore, we extend ISL to a self-supervised learning setting, where accurate causal structure discovery does not rely on any labels. Self-supervised ISL utilizes proposals for invariant causality, by iteratively setting different nodes as targets. On synthetic and real-world datasets, we demonstrate that ISL accurately discovers the causal structure, outperforms alternative methods, and yields superior generalization for datasets with significant distribution shifts.

PDF Details

TMLR Journal 2023 Journal Article

Lightweight Learner for Shared Knowledge Lifelong Learning

  • Yunhao Ge
  • Yuecheng Li
  • Di Wu
  • Ao Xu
  • Adam M. Jones
  • Amanda Sofie Rios
  • Iordanis Fostiropoulos
  • shixian wen

In Lifelong Learning (LL), agents continually learn as they encounter new conditions and tasks. Most current LL is limited to a single agent that learns tasks sequentially. Dedicated LL machinery is then deployed to mitigate the forgetting of old tasks as new tasks are learned. This is inherently slow. We propose a new Shared Knowledge Lifelong Learning (SKILL) challenge, which deploys a decentralized population of LL agents that each sequentially learn different tasks, with all agents operating independently and in parallel. After learning their respective tasks, agents share and consolidate their knowledge over a decentralized communication network, so that, in the end, all agents can master all tasks. We present one solution to SKILL which uses Lightweight Lifelong Learning (LLL) agents, where the goal is to facilitate efficient sharing by minimizing the fraction of the agent that is specialized for any given task. Each LLL agent thus consists of a common task-agnostic immutable part, where most parameters are, and individual task-specific modules that contain fewer parameters but are adapted to each task. Agents share their task-specific modules, plus summary information ("task anchors") representing their tasks in the common task-agnostic latent space of all agents. Receiving agents register each received task-specific module using the corresponding anchor. Thus, every agent improves its ability to solve new tasks each time new task-specific modules and anchors are received. If all agents can communicate with all others, eventually all agents become identical and can solve all tasks. On a new, very challenging SKILL-102 dataset with 102 image classification tasks (5,033 classes in total, 2,041,225 training, 243,464 validation, and 243,464 test images), we achieve much higher (and SOTA) accuracy over 8 LL baselines, while also achieving near perfect parallelization. Code and data can be found at https://github.com/gyhandy/Shared-Knowledge-Lifelong-Learning

PDF Details

NeurIPS Conference 2023 Conference Paper

RoboCLIP: One Demonstration is Enough to Learn Robot Policies

  • Sumedh Sontakke
  • Jesse Zhang
  • Séb Arnold
  • Karl Pertsch
  • Erdem Bıyık
  • Dorsa Sadigh
  • Chelsea Finn
  • Laurent Itti

Reward specification is a notoriously difficult problem in reinforcement learning, requiring extensive expert supervision to design robust reward functions. Imitation learning (IL) methods attempt to circumvent these problems by utilizing expert demonstrations instead of using an extrinsic reward function but typically require a large number of in-domain expert demonstrations. Inspired by advances in the field of Video-and-Language Models (VLMs), we present RoboCLIP, an online imitation learning method that uses a single demonstration (overcoming the large data requirement) in the form of a video demonstration or a textual description of the task to generate rewards without manual reward function design. Additionally, RoboCLIP can also utilize out-of-domain demonstrations, like videos of humans solving the task for reward generation, circumventing the need to have the same demonstration and deployment domains. RoboCLIP utilizes pretrained VLMs without any finetuning for reward generation. Reinforcement learning agents trained with RoboCLIP rewards demonstrate 2-3 times higher zero-shot performance than competing imitation learning methods on downstream robot manipulation tasks, doing so using only one video/text demonstration. Visit our website at https: //sites. google. com/view/roboclip/home for experiment videos.

PDF Details

ICML Conference 2021 Conference Paper

Causal Curiosity: RL Agents Discovering Self-supervised Experiments for Causal Representation Learning

  • Sumedh A. Sontakke
  • Arash Mehrjou
  • Laurent Itti
  • Bernhard Schölkopf

Humans show an innate ability to learn the regularities of the world through interaction. By performing experiments in our environment, we are able to discern the causal factors of variation and infer how they affect the dynamics of our world. Analogously, here we attempt to equip reinforcement learning agents with the ability to perform experiments that facilitate a categorization of the rolled-out trajectories, and to subsequently infer the causal factors of the environment in a hierarchical manner. We introduce a novel intrinsic reward, called causal curiosity, and show that it allows our agents to learn optimal sequences of actions, and to discover causal factors in the dynamics. The learned behavior allows the agent to infer a binary quantized representation for the ground-truth causal factors in every environment. Additionally, we find that these experimental behaviors are semantically meaningful (e. g. , to differentiate between heavy and light blocks, our agents learn to lift them), and are learnt in a self-supervised manner with approximately 2. 5 times less data than conventional supervised planners. We show that these behaviors can be re-purposed and fine-tuned (e. g. , from lifting to pushing or other downstream tasks). Finally, we show that the knowledge of causal factor representations aids zero-shot learning for more complex tasks.

Details

ICRA Conference 2021 Conference Paper

Shaped Policy Search for Evolutionary Strategies using Waypoints *

  • Kiran Lekkala
  • Laurent Itti

In this paper, we try to improve exploration in Blackbox methods, particularly Evolution strategies (ES), when applied to Reinforcement Learning (RL) problems where intermediate waypoints/subgoals are available. Since Evolutionary strategies are highly parallelizable, instead of extracting just a scalar cumulative reward, we use the state-action pairs from the trajectories obtained during rollouts/evaluations, to learn the dynamics of the agent. The learnt dynamics are then used in the optimization procedure to speed-up training. Lastly, we show how our proposed approach is universally applicable by presenting results from experiments conducted on Carla driving and UR5 robotic arm simulators.

Details

ICLR Conference 2021 Conference Paper

Zero-shot Synthesis with Group-Supervised Learning

  • Yunhao Ge
  • Sami Abu-El-Haija
  • Gan Xin
  • Laurent Itti

Visual cognition of primates is superior to that of artificial neural networks in its ability to “envision” a visual object, even a newly-introduced one, in different attributes including pose, position, color, texture, etc. To aid neural networks to envision objects with different attributes, we propose a family of objective functions, expressed on groups of examples, as a novel learning framework that we term Group-Supervised Learning (GSL). GSL allows us to decompose inputs into a disentangled representation with swappable components, that can be recombined to synthesize new samples. For instance, images of red boats & blue cars can be decomposed and recombined to synthesize novel images of red cars. We propose an implementation based on auto-encoder, termed group-supervised zero-shot synthesis network (GZS-Net) trained with our learning framework, that can produce a high-quality red car even if no such example is witnessed during training. We test our model and learning framework on existing benchmarks, in addition to a new dataset that we open-source. We qualitatively and quantitatively demonstrate that GZS-Net trained with GSL outperforms state-of-the-art methods

Details

IJCAI Conference 2019 Conference Paper

Closed-Loop Memory GAN for Continual Learning

  • Amanda Rios
  • Laurent Itti

Sequential learning of tasks using gradient descent leads to an unremitting decline in the accuracy of tasks for which training data is no longer available, termed catastrophic forgetting. Generative models have been explored as a means to approximate the distribution of old tasks and bypass storage of real data. Here we propose a cumulative closed-loop memory replay GAN (CloGAN) provided with external regularization by a small memory unit selected for maximum sample diversity. We evaluate incremental class learning using a notoriously hard paradigm, single-headed learning, in which each task is a disjoint subset of classes in the overall dataset, and performance is evaluated on all previous classes. First, we show that when constructing a dynamic memory unit to preserve sample heterogeneity, model performance asymptotically approaches training on the full dataset. We then show that using a stochastic generator to continuously output fresh new images during training increases performance significantly further meanwhile generating quality images. We compare our approach to several baselines including fine-tuning by gradient descent (FGD), Elastic Weight Consolidation (EWC), Deep Generative Replay (DGR) and Memory Replay GAN (MeRGAN). Our method has very low long-term memory cost, the memory unit, as well as negligible intermediate memory storage.

PDF Details

IROS Conference 2019 Conference Paper

Inertial-based Motion Capturing and Smart Training System

  • Jens Windau
  • Laurent Itti

Smart coaching platforms are emerging which combine Body-Sensor-Networks with AI-based training software to monitor and analyze body motions of athletes, workers, or medical patients. This allows for new opportunities to explore algorithms to interpret body sensor data and provide analytical feedback for learning a physical task, refining body motions, or to protect from work-related injuries. This paper presents a solution to non-invasively equip a person with sensors of a Smart Training System (STS) to improve training efficiency during sport activities. Our system calculates the significance of each body part during physical activities and provides targeted feedback on which body locations are under-performing. In experiments, the system collected data from 13 inertial sensors attached to the entire body of inexperienced golf learners. Using an indoor golf training net with a central target with 3 concentric zones, 1, 080 real-world golf swings of 11 participants were analyzed. During the first 30 swings of each participant, the system learned distributions of motions from each sensor, conditioned on swing performance reported by users from their hitting location on the target. In the later 70 swings, feedback was provided to a subgroup of 8 participants, by computing, for an optimal set of features determined during training, the largest discrepancy. The remaining 3 (control) participants received no feedback. From only 100 golf swings for each participant, our system led to significantly improved scores by on average 3. 7x (t-test, $\mathrm{p}\lt 0. 0001)$ over the latter 70 swings. Our results suggest that the combination of motion sensors and processing developed here was able to yield significantly improved golf swing training.

Details

ICML Conference 2018 Conference Paper

Born-Again Neural Networks

  • Tommaso Furlanello
  • Zachary C. Lipton
  • Michael Tschannen
  • Laurent Itti
  • Anima Anandkumar

Knowledge Distillation (KD) consists of transferring “knowledge” from one machine learning model (the teacher) to another (the student). Commonly, the teacher is a high-capacity model with formidable performance, while the student is more compact. By transferring knowledge, one hopes to benefit from the student’s compactness, without sacrificing too much performance. We study KD from a new perspective: rather than compressing models, we train students parameterized identically to their teachers. Surprisingly, these Born-Again Networks (BANs), outperform their teachers significantly, both on computer vision and language modeling tasks. Our experiments with BANs based on DenseNets demonstrate state-of-the-art performance on the CIFAR-10 (3. 5%) and CIFAR-100 (15. 5%) datasets, by validation error. Additional experiments explore two distillation objectives: (i) Confidence-Weighted by Teacher Max (CWTM) and (ii) Dark Knowledge with Permuted Predictions (DKPP). Both methods elucidate the essential components of KD, demonstrating the effect of the teacher outputs on both predicted and non-predicted classes.

Details

ICRA Conference 2018 Conference Paper

DeepVP: Deep Learning for Vanishing Point Detection on 1 Million Street View Images

  • Chin-Kai Chang
  • Jiaping Zhao
  • Laurent Itti

We propose a novel approach to detect vanishing points in images using a convolutional neural network (CNN) trained on a newly collected Google street-view image dataset. By utilizing the camera parameters and road direction data from Google street view, we collected a total of 1, 053, 425 images with inferred ground-truth vanishing points, along 23 worldwide routes totaling 125, 165 kilometers. We then formulate vanishing point detection as a CNN classification problem using an output layer with 225 discrete possible vanishing point locations. Experimental results show that our deep vanishing point system outperforms the state-of-the-art algorithmic vanishing point detector. We achieved 99% accuracy in recovering the horizon line and 92% in locating the vanishing point within a ±5-degree range.

Details

ICRA Conference 2018 Conference Paper

Inertial Machine Monitoring System for Automated Failure Detection

  • Jens Windau
  • Laurent Itti

Smart manufacturing technologies are emerging which combine industrial equipment with Internet-of-Things (IoT) sensors to monitor and improve productivity of manufacturing. This allows for new opportunities to explore algorithms for predicting machine failures from attached sensor data. This paper presents a solution to non-invasively upgrade an existing machine with an Inertial Machine Monitoring System (IMMS) to detect and classify equipment failure or degraded state. We also provide a strategy to optimize the amount, placement locations, and efficiency of the sensors. In experiments, the system collected data from 36 inertial sensors placed at multiple locations on a 3D printer. Normal operation vs. 10 types of realworld abnormal equipment behavior (loose belt, failures of machine components) were detected and classified by Support Vector Machines and Neural Networks. Using under 1 minute of recording while running a test print, a recursively discovered best subset of 4 to 9 sensors yielded 11-way classification accuracy over 99%. Our results suggest that even a small sensor network and short test program can yield effective detection of machine degraded state and can facilitate early remediation.

Details

ICRA Conference 2016 Conference Paper

Walking compass with head-mounted IMU sensor

  • Jens Windau
  • Laurent Itti

Emerging wearable technologies offer new sensor placement options on the human body. Particularly, head-mounted glass-wear opens up new data capturing possibilities directly from the human head. This allows exploring new cyber-robotics algorithms (robotics sensors and human motor plant). Glass-wear systems, however, require additional compensation for head motions that will affect the captured sensor data. Particularly, pedestrian dead-reckoning (PDR), activity recognition, and other applications are limited or restricted when head-mounted sensors are used, because of possible confusion between head and body movements. Thus, previous PDR approaches typically required to keep the head pointing direction aligned with the walking direction to avoid positional errors. This paper presents a head-mounted orientation system (HOS) that identifies and filters out interfering head motions in 3 steps. Step 1 transforms inertial sensor data into a stable normalized coordinate system (roll/pitch motion compensated). Step 2 compares walking patterns before and after a rotating motion. Step 3 eliminates interfering head motions from sensor data by dynamically adjusting the noise parameters of the extended Kalman filter. HOS has been implemented on a Google Glass platform and achieved high accuracy in tracking a person's path even in the presence of head movements (within 2. 5% of traveled distance) when tested in multiple real-world scenarios. By eliminating head motions, HOS not only enables accurate PDR, but also facilitates the task for downstream activity recognition algorithms.

Details

NeurIPS Conference 2013 Conference Paper

Bayesian optimization explains human active search

  • Ali Borji
  • Laurent Itti

Many real-world problems have complicated objective functions. To optimize such functions, humans utilize sophisticated sequential decision-making strategies. Many optimization algorithms have also been developed for this same purpose, but how do they compare to humans in terms of both performance and behavior? We try to unravel the general underlying algorithm people may be using while searching for the maximum of an invisible 1D function. Subjects click on a blank screen and are shown the ordinate of the function at each clicked abscissa location. Their task is to find the function’s maximum in as few clicks as possible. Subjects win if they get close enough to the maximum location. Analysis over 23 non-maths undergraduates, optimizing 25 functions from different families, shows that humans outperform 24 well-known optimization algorithms. Bayesian Optimization based on Gaussian Processes, which exploit all the x values tried and all the f(x) values obtained so far to pick the next x, predicts human performance and searched locations better. In 6 follow-up controlled experiments over 76 subjects, covering interpolation, extrapolation, and optimization tasks, we further confirm that Gaussian Processes provide a general and unified theoretical account to explain passive and active function learning and search in humans.

PDF Details

IROS Conference 2013 Conference Paper

Beobot 2. 0: Autonomous mobile robot localization and navigation in outdoor pedestrian environment

  • Chin-Kai Chang
  • Christian Siagian
  • Laurent Itti

We present Beobot 2. 0 [1], an autonomous mobile robot designed to operate in unconstrained urban environments. The goal of the project is to create service robots that can be deployed for various tasks that require long range travel. Over the past two years, Beobot has successfully traversed various paths across the USC campus, demonstrating its robustness in recognizing and following different types of roads, avoiding obstacles such as pedestrians and service vehicles, and finding its way to the goal.

Details

IROS Conference 2013 Conference Paper

Finding planes in LiDAR point clouds for real-time registration

  • W. Shane Grant
  • Randolph Voorhies
  • Laurent Itti

We present a robust plane finding algorithm that when combined with plane-based frame-to-frame registration gives accurate real-time pose estimation. Our plane extraction is capable of handling large and sparse datasets such as those generated from spinning multi-laser sensors such as the Velodyne HDL-32E LiDAR. We test our algorithm on frame-to-frame registration in a closed-loop indoor path comprising 827 successive 3D laser scans (over 57 million points), using no additional information (e. g. , odometry, IMU). Our algorithm outperforms, in both accuracy and time, three state-of-the-art methods, based on iterative closest point (ICP), plane-based randomized Hough transform, and planar region growing.

Details

ICRA Conference 2013 Conference Paper

Mobile robot navigation system in outdoor pedestrian environment using vision-based road recognition

  • Christian Siagian
  • Chin-Kai Chang
  • Laurent Itti

We present a mobile robot navigation system guided by a novel vision-based road recognition approach. The system represents the road as a set of lines extrapolated from the detected image contour segments. These lines enable the robot to maintain its heading by centering the vanishing point in its field of view, and to correct the long term drift from its original lateral position. We integrate odometry and our visual road recognition system into a grid-based local map that estimates the robot pose as well as its surroundings to generate a movement path. Our road recognition system is able to estimate the road center on a standard dataset with 25, 076 images to within 11. 42 cm (with respect to roads at least 3 m wide). It outperforms three other state-of-the-art systems. In addition, we extensively test our navigation system in four busy college campus environments using a wheeled robot. Our tests cover more than 5 km of autonomous driving without failure. This demonstrates robustness of the proposed approach against challenges that include occlusion by pedestrians, non-standard complex road markings and shapes, shadows, and miscellaneous obstacle objects.

Details

IROS Conference 2013 Conference Paper

Situation awareness via sensor-equipped eyeglasses

  • Jens Windau
  • Laurent Itti

New smartphone technologies are emerging which combine head-mounted displays (HMD) with standard functions such as receiving phone calls, emails, and helping with navigation. This opens new opportunities to explore cyber robotics algorithms (robotics sensors and human motor plant). To make these devices more adaptive to the environmental conditions, user behavior, and user preferences, it is important to allow the sensor-equipped devices to efficiently adapt and respond to user activities (e. g. , disable incoming phone calls in an elevator, activate video recording while car driving). This paper hence presents a situation awareness system (SAS) for head-mounted smartphones. After collecting data from inertial sensors (accelerometers, gyroscopes), and video data (camera), SAS performs activity classification in three steps. Step 1 transforms inertial sensor data into a head orientation-independent and stable normalized coordinate system. Step 2 extracts critical features (statistical, physical, GIST). Step 3 classifies activities (Naive Bayes classifier), distinguishes between environments (Support Vector Machine), and finally combines both results (Hidden Markov Model) for further improvement. SAS has been implemented on a sensor-equipped eyeglasses prototype and achieved high accuracy (81. 5%) when distinguishing between 20 real-world activities.

Details

AAAI Conference 2012 Conference Paper

An Object-Based Bayesian Framework for Top-Down Visual Attention

  • Ali Borji
  • Dicky Sihite
  • Laurent Itti

We introduce a new task-independent framework to model top-down overt visual attention based on graphical models for probabilistic inference and reasoning. We describe a Dynamic Bayesian Network (DBN) that infers probability distributions over attended objects and spatial locations directly from observed data. Probabilistic inference in our model is performed over object-related functions which are fed from manual annotations of objects in video scenes or by state-of-theart object detection models. Evaluating over ∼3 hours (appx. 315, 000 eye fixations and 12, 600 saccades) of observers playing 3 video games (time-scheduling, driving, and flight combat), we show that our approach is significantly more predictive of eye fixations compared to: 1) simpler classifier-based models also developed here that map a signature of a scene (multi-modal information from gist, bottom-up saliency, physical actions, and events) to eye positions, 2) 14 state-of-theart bottom-up saliency models, and 3) brute-force algorithms such as mean eye position. Our results show that the proposed model is more effective in employing and reasoning over spatio-temporal visual data.

PDF Details

IROS Conference 2012 Conference Paper

Mobile robot monocular vision navigation based on road region and boundary estimation

  • Chin-Kai Chang
  • Christian Siagian
  • Laurent Itti

We present a monocular vision-based navigation system that incorporates two contrasting approaches: region segmentation that computes the road appearance, and road boundary detection that estimates the road shape. The former approach segments the image into multiple regions, then selects and tracks the most likely road appearance. On the other hand, the latter detects the vanishing point and road boundaries to estimate the shape of the road. Our algorithm operates in urban road settings and requires no training or camera calibration to maximize its adaptability to many environments. We tested our system in 1 indoor and 3 outdoor urban environments using our ground-based robot, Beobot 2. 0, for real-time autonomous visual navigation. In 20 trial runs the robot was able to travel autonomously for 98. 19% of the total route length of 316. 60m.

Details

ICRA Conference 2012 Conference Paper

Modeling the influence of action on spatial attention in visual interactive environments

  • Ali Borji
  • Dicky N. Sihite
  • Laurent Itti

A large number of studies have been reported on top-down influences of visual attention. However, less progress have been made in understanding and modeling its mechanisms in real-world tasks. In this paper, we propose an approach for learning spatial attention taking into account influences of physical actions on top-down attention. For this purpose, we focus on interactive visual environments (video games) which are modest real-world simulations, where a player has to attend to certain aspects of visual stimuli and perform actions to achieve a goal. The basic idea is to learn a mapping from current mental state of the game player, represented by past actions and observations, to its gaze fixation. A data-driven approach is followed where we train a model from the data of some players and test it over a new subject. In particular, two contributions this paper makes are: 1) employing multi-modal information including mean eye position, gist of a scene, physical actions, bottom-up saliency, and tagged events for state representation and 2) analysis of different methods of combining bottom-up and top-down influences. Comparing with other top-down task-driven and bottom-up spatio-temporal models, our approach shows higher NSS scores in predicting eye positions.

Details

IROS Conference 2011 Conference Paper

Multilayer real-time video image stabilization

  • Jens Windau
  • Laurent Itti

In many camera-based robotics applications, stabilizing video images in real-time is often critical for successful performance. In particular vision-based navigation, localization and tracking tasks cannot be performed reliably when landmarks are blurry, poorly focused or disappear from the camera view due to strong vibrations. Thus a reliable video image stabilization system would be invaluable for these applications. This paper presents a real-time video image stabilization system (VISS) primarily developed for aerial robots. Its unique architecture combines four independent stabilization layers. Layer 1 detects vibrations via an inertial measurement unit (IMU) and performs external counter-movements with a motorized gimbal. Layer 2 damps vibrations by using mechanical devices. The internal optical image stabilization of the camera represents Layer 3, while Layer 4 filters remaining vibrations using software. VISS is low-cost and robust. It has been implemented on a “Photoship One” gimbal, using GUMBOT hardware for processing Sparkfun-IMU data (Layer 1). Lord Mount vibration isolators damp vibrations (Layer 2). Video images of Panasonic's Lumix DMCTZ5 camera are optically stabilized with Panasonic's “Mega O. I. S. ” technique (Layer 3) and digitally stabilized with “Deshaker” software (Layer 4). VISS significantly improved the stability of shaky video images in a series of experiments.

Details

ICRA Conference 2011 Conference Paper

Scene classification with a sparse set of salient regions

  • Ali Borji
  • Laurent Itti

This work proposes an approach for scene classification by extracting and matching visual features only at the focuses of visual attention instead of the entire scene. Analysis over a database of natural scenes demonstrates that regions proposed by the saliency-based model of visual attention are robust to image transformations. Using a nearest neighbor classifier and a distance measure defined over the salient regions, we obtained 97. 35% and 78. 28% classification rates with SIFT and C2 features from the HMAX model at 5 salient regions covering at most 31% of the image. Classification with features extracted from the entire image results in 99. 3% and 82. 32% using SIFT and C2 features, respectively. Comparing attentional and adhoc approaches shows that classification rate of the first approach is 0. 95 of the second. Overall, our results prove that efficient scene classification, in terms of reducing the complexity of feature extraction is possible without a significant drop in performance.

Details

IROS Conference 2010 Conference Paper

Mobile robot vision navigation & localization using Gist and Saliency

  • Chin-Kai Chang
  • Christian Siagian
  • Laurent Itti

We present a vision-based navigation and localization system using two biologically-inspired scene understanding models which are studied from human visual capabilities: (1) Gist model which captures the holistic characteristics and layout of an image and (2) Saliency model which emulates the visual attention of primates to identify conspicuous regions in the image. Here the localization system utilizes the gist features and salient regions to accurately localize the robot, while the navigation system uses the salient regions to perform visual feedback control to direct its heading and go to a user-provided goal location. We tested the system on our robot, Beobot2. 0, in an indoor and outdoor environment with a route length of 36. 67m (10, 890 video frames) and 138. 27m (28, 971 frames), respectively. On average, the robot is able to drive within 3. 68cm and 8. 78cm (respectively) of the center of the lane.

Details

IROS Conference 2009 Conference Paper

Centralized server environment for educational robotics

  • Randolph Voorhies
  • Christian Siagian
  • Lior Elazary
  • Laurent Itti

One of the main challenges when creating an undergraduate introduction to robotics course is connecting the theory taught in the lectures with the current practices of research. The primary cause of this difficulty is an inability to find a hardware solution that is powerful enough to run complex cutting-edge algorithms yet inexpensive enough to be purchased by an undergraduate class budget. An ideal system needs to have a gentle learning curve to allow students with minimal background in the field to get a robot up and running. Lastly, a fleet of classroom robots needs to be easy to administrate and maintain given the limited time of a Teaching Assistant. Our approach is to implement a centralized server system. In this system individual robots are inexpensive yet capable of establishing a WiFi link to a main server so that all the compilation and system administration, as well as much of the computationally intensive processing, are done on that server. We find that this solution saves both time and money and provides an effective teaching tool. This paper describes the hardware and software architecture of the system, and example applications implemented by undergraduate students.

Details

ICRA Conference 2008 Conference Paper

Storing and recalling information for vision localization

  • Christian Siagian
  • Laurent Itti

In implementing a vision localization system, a crucial issue to consider is how to efficiently store and recall the necessary information so that the robot is not only able to accurately localize itself, but does so in a timely manner. In the presented system, we discuss a strategy to minimize the amount of stored data by analyzing the strengths and weaknesses of several cooperating recognition modules, and by using them through a prioritization scheme, which orders the data entries from the most likely to match to the least. We validate the system is a series of experiments at three large scale outdoor environments: a building complex (126 times 180 ft. area, 3583 testing images), a vegetation-filled park (270 times 360 ft. area, 6006 testing images), and an open-field area (450 times 585 ft. area, 8823 testing images) - each with its own set of challenges. Not only is the system able to localize in these environments (on average 3. 46 ft. , 6. 55 ft. 12. 96 ft. of error, respectively), it does so while searching through only 7. 35%, 3. 50%, and 6. 12% of all the stored information, respectively.

Details

IROS Conference 2007 Conference Paper

Biologically-inspired robotics vision monte-carlo localization in the outdoor environment

  • Christian Siagian
  • Laurent Itti

We present a robot localization system using biologically-inspired vision. Our system models two extensively studied human visual capabilities: (1) extracting the "gist" of a scene to produce a coarse localization hypothesis, and (2) refining it by locating salient landmark regions in the scene. Gist is computed here as a holistic statistical signature of the image, yielding abstract scene classification and layout. Saliency is computed as a measure of interest at every image location, efficiently directing the time-consuming landmark identification process towards the most likely candidate locations in the image. The gist and salient landmark features are then further processed using a Monte-Carlo localization algorithm to allow the robot to generate its position. We test the system in three different outdoor environments - building complex (126times180 ft. area, 3794 testing images), vegetation-filled park (270times360 ft. area, 7196 testing images), and open-field park (450times585 ft. area, 8287 testing images) - each with its own challenges. The system is able to localize, on average, within 6. 0, 10. 73, and 32. 24 ft. , respectively, even with multiple kidnapped-robot instances.

Details

NeurIPS Conference 2007 Conference Paper

Congruence between model and human attention reveals unique signatures of critical visual events

  • Robert Peters
  • Laurent Itti

Current computational models of bottom-up and top-down components of atten- tion are predictive of eye movements across a range of stimuli and of simple, fixed visual tasks (such as visual search for a target among distractors). How- ever, to date there exists no computational framework which can reliably mimic human gaze behavior in more complex environments and tasks, such as driving a vehicle through traffic. Here, we develop a hybrid computational/behavioral framework, combining simple models for bottom-up salience and top-down rel- evance, and looking for changes in the predictive power of these components at different critical event times during 4. 7 hours (500, 000 video frames) of observers playing car racing and flight combat video games. This approach is motivated by our observation that the predictive strengths of the salience and relevance mod- els exhibit reliable temporal signatures during critical event windows in the task sequence—for example, when the game player directly engages an enemy plane in a flight combat game, the predictive strength of the salience model increases significantly, while that of the relevance model decreases significantly. Our new framework combines these temporal signatures to implement several event detec- tors. Critically, we find that an event detector based on fused behavioral and stim- ulus information (in the form of the model’s predictive strength) is much stronger than detectors based on behavioral information alone (eye position) or image in- formation alone (model prediction maps). This approach to event detection, based on eye tracking combined with computational models applied to the visual input, may have useful applications as a less-invasive alternative to other event detection approaches based on neural signatures derived from EEG or fMRI recordings.

PDF Details

NeurIPS Conference 2005 Conference Paper

Bayesian Surprise Attracts Human Attention

  • Laurent Itti
  • Pierre Baldi

The concept of surprise is central to sensory processing, adaptation, learning, and attention. Yet, no widely-accepted mathematical theory currently exists to quantitatively characterize surprise elicited by a stimulus or event, for observers that range from single neurons to complex natural or engineered systems. We describe a formal Bayesian definition of surprise that is the only consistent formulation under minimal axiomatic assumptions. Surprise quantifies how data affects a natural or artificial observer, by measuring the difference between posterior and prior beliefs of the observer. Using this framework we measure the extent to which humans direct their gaze towards surprising items while watching television and video games. We find that subjects are strongly attracted towards surprising locations, with 72% of all human gaze shifts directed towards locations more surprising than the average, a figure which rises to 84% when considering only gaze targets simultaneously selected by all subjects. The resulting theory of surprise is applicable across different spatio-temporal scales, modalities, and levels of abstraction. Life is full of surprises, ranging from a great christmas gift or a new magic trick, to wardrobe malfunctions, reckless drivers, terrorist attacks, and tsunami waves. Key to survival is our ability to rapidly attend to, identify, and learn from surprising events, to decide on present and future courses of action [1]. Yet, little theoretical and computational understanding exists of the very essence of surprise, as evidenced by the absence from our everyday vocabulary of a quantitative unit of surprise: Qualities such as the "wow factor" have remained vague and elusive to mathematical analysis. Informal correlates of surprise exist at nearly all stages of neural processing. In sensory neuroscience, it has been suggested that only the unexpected at one stage is transmitted to the next stage [2]. Hence, sensory cortex may have evolved to adapt to, to predict, and to quiet down the expected statistical regularities of the world [3, 4, 5, 6], focusing instead on events that are unpredictable or surprising. Electrophysiological evidence for this early sensory emphasis onto surprising stimuli exists from studies of adaptation in visual [7, 8, 4, 9], olfactory [10, 11], and auditory cortices [12], subcortical structures like the LGN [13], and even retinal ganglion cells [14, 15] and cochlear hair cells [16]: neural response greatly attenuates with repeated or prolonged exposure to an initially novel stimulus. Surprise and novelty are also central to learning and memory formation [1], to the point that surprise is believed to be a necessary trigger for associative learning [17, 18], as supported by mounting evidence for a role of the hippocampus as a novelty detector [19, 20, 21]. Finally, seeking novelty is a well-identified human character trait, with possible association with the dopamine D4 receptor gene [22, 23, 24]. In the Bayesian framework, we develop the only consistent theory of surprise, in terms of the difference between the posterior and prior distributions of beliefs of an observer over the available class of models or hypotheses about the world. We show that this definition derived from first principles presents key advantages over more ad-hoc formulations, typically relying on detecting outlier stimuli. Armed with this new framework, we provide direct experimental evidence that surprise best characterizes what attracts human gaze in large amounts of natural video stimuli. We here extend a recent pilot study [25], adding more comprehensive theory, large-scale human data collection, and additional analysis.

PDF Details

NeurIPS Conference 2005 Conference Paper

Optimal cue selection strategy

  • Vidhya Navalpakkam
  • Laurent Itti

Survival in the natural world demands the selection of relevant visual cues to rapidly and reliably guide attention towards prey an d predators in cluttered environments. We investigate whether our visu al system selects cues that guide search in an optimal manner. We formall y obtain the optimal cue selection strategy by maximizing the signal to noise ratio (S N R) between a search target and surrounding distractors. This optimal strategy successfully accounts for several phenom ena in visual search behavior, including the effect of target-distracto r discriminability, uncertainty in target's features, distractor heterogenei ty, and linear separability. Furthermore, the theory generates a new predict ion, which we verify through psychophysical experiments with human subj ects. Our results provide direct experimental evidence that humans sel ect visual cues so as to maximize S N R between the targets and surrounding clutter.

PDF Details

NeurIPS Conference 2001 Conference Paper

Modeling the Modulatory Effect of Attention on Human Spatial Vision

  • Laurent Itti
  • Jochen Braun
  • Christof Koch

We present new simulation results, in which a computational model of interacting visual neurons simultaneously predicts the modula(cid: 173) tion of spatial vision thresholds by focal visual attention, for five dual-task human psychophysics experiments. This new study com(cid: 173) plements our previous findings that attention activates a winner(cid: 173) take-all competition among early visual neurons within one cortical hypercolumn. This "intensified competition" hypothesis assumed that attention equally affects all neurons, and yielded two single(cid: 173) unit predictions: an increase in gain and a sharpening of tuning with attention. While both effects have been separately observed in electrophysiology, no single-unit study has yet shown them si(cid: 173) multaneously. Hence, we here explore whether our model could still predict our data if attention might only modulate neuronal gain, but do so non-uniformly across neurons and tasks. Specifically, we investigate whether modulating the gain of only the neurons that are loudest, best-tuned, or most informative about the stimulus, or of all neurons equally but in a task-dependent manner, may ac(cid: 173) count for the data. We find that none of these hypotheses yields predictions as plausible as the intensified competition hypothesis, hence providing additional support for our original findings.

PDF Details

NeurIPS Conference 1998 Conference Paper

Attentional Modulation of Human Pattern Discrimination Psychophysics Reproduced by a Quantitative Model

  • Laurent Itti
  • Jochen Braun
  • Dale Lee
  • Christof Koch

We previously proposed a quantitative model of early visual pro(cid: 173) cessing in primates, based on non-linearly interacting visual filters and statistically efficient decision. We now use this model to inter(cid: 173) pret the observed modulation of a range of human psychophysical thresholds with and without focal visual attention. Our model - calibrated by an automatic fitting procedure - simultaneously re(cid: 173) produces thresholds for four classical pattern discrimination tasks, performed while attention was engaged by another concurrent task. Our model then predicts that the seemingly complex improvements of certain thresholds, which we observed when attention was fully available for the discrimination tasks, can best be explained by a strengthening of competition among early visual filters.

PDF Details

NeurIPS Conference 1997 Conference Paper

A Model of Early Visual Processing

  • Laurent Itti
  • Jochen Braun
  • Dale Lee
  • Christof Koch

We propose a model for early visual processing in primates. The model consists of a population of linear spatial filters which inter(cid: 173) act through non-linear excitatory and inhibitory pooling. Statisti(cid: 173) cal estimation theory is then used to derive human psychophysical thresholds from the responses of the entire population of units. The model is able to reproduce human thresholds for contrast and ori(cid: 173) entation discrimination tasks, and to predict contrast thresholds in the presence of masks of varying orientation and spatial frequency.

PDF Details
v2026.09.13