Arrow Research search

Author name cluster

Robert Sim

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

23 papers
2 author rows

Possible papers

23

NeurIPS Conference 2025 Conference Paper

Contextual Integrity in LLMs via Reasoning and Reinforcement Learning

  • Guangchen (Eric) Lan
  • Huseyin A. Inan
  • Sahar Abdelnabi
  • Janardhan Kulkarni
  • Lukas Wutschitz
  • Reza Shokri
  • Christopher Brinton
  • Robert Sim

As the era of autonomous agents making decisions on behalf of users unfolds, ensuring contextual integrity (CI) -- what is the appropriate information to share while carrying out a certain task -- becomes a central question to the field. We posit that CI demands a form of reasoning where the agent needs to reason about the context in which it is operating. To test this, we first prompt LLMs to reason explicitly about CI when deciding what information to disclose. We then extend this approach by developing a reinforcement learning (RL) framework that further instills in models the reasoning necessary to achieve CI. Using a synthetic, automatically created, dataset of only $\sim700$ examples but with diverse contexts and information disclosure norms, we show that our method substantially reduces inappropriate information disclosure while maintaining task performance across multiple model sizes and families. Importantly, improvements transfer from this synthetic dataset to established CI benchmarks such as PrivacyLens that has human annotations and evaluates privacy leakage of AI assistants in actions and tool calls.

PDF Details

NeurIPS Conference 2025 Conference Paper

Learning to Specialize: Joint Gating-Expert Training for Adaptive MoEs in Decentralized Settings

  • Yehya Farhat
  • Hamza ElMokhtar Shili
  • Fangshuo Liao
  • Chen Dun
  • Mirian Hipolito Garcia
  • Guoqing Zheng
  • Ahmed Awadallah
  • Robert Sim

Mixture-of-Experts (MoEs) achieve scalability by dynamically activating subsets of their components. Yet, understanding how expertise emerges through joint training of gating mechanisms and experts remains incomplete, especially in scenarios without clear task partitions. Motivated by inference costs and data heterogeneity, we study how joint training of gating functions and experts can dynamically allocate domain-specific expertise across multiple underlying data distributions. As an outcome of our framework, we develop an instance tailored specifically to decentralized training scenarios, introducing Dynamically Decentralized Orchestration of MoEs or DDOME. DDOME leverages heterogeneity emerging from distributional shifts across decentralized data sources to specialize experts dynamically. By integrating a pretrained common expert to inform a gating function, DDOME achieves personalized expert subset selection on-the-fly, facilitating just-in-time personalization. We empirically validate DDOME within a Federated Learning (FL) context: DDOME attains from 4\% up to an 24\% accuracy improvement over state-of-the-art FL baselines in image and text classification tasks, while maintaining competitive zero-shot generalization capabilities. Furthermore, we provide theoretical insights confirming that the joint gating-experts training is critical for achieving meaningful expert specialization.

PDF Details

AAAI Conference 2025 Conference Paper

Sweeping Heterogeneity with Smart MoPs: Mixture of Prompts for LLM Task Adaptation

  • Chen Dun
  • Mirian Del Carmen Hipolito Garcia
  • Guoqing Zheng
  • Ahmed Hassan Awadallah
  • Robert Sim
  • Anastasios Kyrillidis

Prompt instruction tuning is a popular approach to better adjust pretrained LLMs for specific downstream tasks. How to extend this approach to simultaneously handle multiple tasks and data distributions is an interesting question. We propose Mixture of Prompts (MoPs) with smart gating functionality. Our proposed system identifies relevant skills embedded in different groups of prompts and dynamically weighs experts (i.e., collection of prompts) based on the target task. Experiments show that MoPs are resilient to model compression, data source, and task composition, making them highly versatile and applicable in various contexts. In practice, MoPs can simultaneously mitigate prompt training ``interference'' in multi-task, multi-source scenarios (e.g., task and data heterogeneity across sources) and possible implications from model approximations. Empirically, MoPs show particular effectiveness in compressed model scenarios, while maintaining favorable performance in uncompressed settings: MoPs can reduce final perplexity from 9% up to 70% in non-i.i.d. distributed cases and from 3% up to 30% in centralized cases, compared to baselines.

PDF Details DOI

ICLR Conference 2024 Conference Paper

Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing

  • Dujian Ding
  • Ankur Mallick
  • Chi Wang 0001
  • Robert Sim
  • Subhabrata Mukherjee
  • Victor Rühle
  • Laks V. S. Lakshmanan
  • Ahmed Hassan Awadallah

Large language models (LLMs) excel in most NLP tasks but also require expensive cloud servers for deployment due to their size, while smaller models that can be deployed on lower cost (e.g., edge) devices, tend to lag behind in terms of response quality. Therefore in this work we propose a hybrid inference approach which combines their respective strengths to save cost and maintain quality. Our approach uses a router that assigns queries to the small or large model based on the predicted query difficulty and the desired quality level. The desired quality level can be tuned dynamically at test time to seamlessly trade quality for cost as per the scenario requirements. In experiments our approach allows us to make up to 40% fewer calls to the large model, with no drop in response quality.

Details

ICLR Conference 2024 Conference Paper

Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generation

  • Xinyu Tang 0003
  • Richard Shin
  • Huseyin A. Inan
  • Andre Manoel
  • Niloofar Mireshghallah
  • Zinan Lin 0001
  • Sivakanth Gopi
  • Janardhan Kulkarni

We study the problem of in-context learning (ICL) with large language models (LLMs) on private datasets. This scenario poses privacy risks, as LLMs may leak or regurgitate the private examples demonstrated in the prompt. We propose a novel algorithm that generates synthetic few-shot demonstrations from the private dataset with formal differential privacy (DP) guarantees, and show empirically that it can achieve effective ICL. We conduct extensive experiments on standard benchmarks and compare our algorithm with non-private ICL and zero-shot solutions. Our results demonstrate that our algorithm can achieve competitive performance with strong privacy levels. These results open up new possibilities for ICL with privacy protection for a broad range of applications.

Details

ICLR Conference 2024 Conference Paper

Privately Aligning Language Models with Reinforcement Learning

  • Fan Wu
  • Huseyin A. Inan
  • Arturs Backurs
  • Varun Chandrasekaran
  • Janardhan Kulkarni
  • Robert Sim

Positioned between pre-training and user deployment, aligning large language models (LLMs) through reinforcement learning (RL) has emerged as a prevailing strategy for training instruction following-models such as ChatGPT. In this work, we initiate the study of privacy-preserving alignment of LLMs through Differential Privacy (DP) in conjunction with RL. Following the influential work of Ziegler et al. (2020), we study two dominant paradigms: (i) alignment via RL without human in the loop (e.g., positive review generation) and (ii) alignment via RL from human feedback (RLHF) (e.g., summarization in a human-preferred way). We give a new DP framework to achieve alignment via RL, and prove its correctness. Our experimental results validate the effectiveness of our approach, offering competitive utility while ensuring strong privacy protections.

Details

IJCAI Conference 2022 Conference Paper

Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learning

  • Yae Jee Cho
  • Andre Manoel
  • Gauri Joshi
  • Robert Sim
  • Dimitrios Dimitriadis

Federated learning (FL) enables edge-devices to collaboratively learn a model without disclosing their private data to a central aggregating server. Most existing FL algorithms require models of identical architecture to be deployed across the clients and server, making it infeasible to train large models due to clients' limited system resources. In this work, we propose a novel ensemble knowledge transfer method named Fed-ET in which small models (different in architecture) are trained on clients, and used to train a larger model at the server. Unlike in conventional ensemble learning, in FL the ensemble can be trained on clients' highly heterogeneous data. Cognizant of this property, Fed-ET uses a weighted consensus distillation scheme with diversity regularization that efficiently extracts reliable consensus from the ensemble while improving generalization by exploiting the diversity within the ensemble. We show the generalization bound for the ensemble of weighted models trained on heterogeneous datasets that supports the intuition of Fed-ET. Our experiments on image and language tasks show that Fed-ET significantly outperforms other state-of-the-art FL algorithms with fewer communicated parameters, and is also robust against high data-heterogeneity.

PDF Details DOI

IROS Conference 2006 Conference Paper

Autonomous vision-based exploration and mapping using hybrid maps and Rao-Blackwellised particle filters

  • Robert Sim
  • James J. Little

This paper addresses the problem of exploring and mapping an unknown environment using a robot equipped with a stereo vision sensor. The main contribution of our work is a fully automatic mapping system that operates without the use of active ranger sensors (such as laser or sonic transducers), can operate in real-time and can consistently produce accurate maps of large-scale environments. Our approach implements a Rao-Blackwellised particle filter (RBPF) to solve the simultaneous localization and mapping problem and uses efficient data structures for real-time data association, mapping, and spatial reasoning. We employ a hybrid map representation that infers 3D point landmarks from image features to achieve precise localization, coupled with occupancy grids for safe navigation. This paper describes our framework and implementation, and presents our exploration method, and experimental results illustrating the functionality of the system

Details

ICRA Conference 2006 Conference Paper

σSLAM: Stereo Vision SLAM using the Rao-Blackwellised Particle Filter and a Novel Mixture Proposal Distribution

  • Pantelis Elinas
  • Robert Sim
  • James J. Little

We consider the problem of simultaneous localization and mapping (SLAM) using the Rao-Blackwellised particle filter (RBPF) for the class of indoor mobile robots equipped only with stereo vision. Our goal is to construct dense metric maps of natural 3D point landmarks for large cyclic environments in the absence of accurate landmark position measurements and motion estimates. Our work differs from other approaches because landmark estimates are derived from stereo vision and motion estimates are based on sparse optical flow. We distinguish between landmarks using the scale invariant feature transform (SIFT). This is in contrast to current popular approaches that rely on reliable motion models derived from odometric hardware and accurate landmark measurements obtained with laser sensors. Since our approach depends on a particle filter whose main component is the proposal distribution, we develop and evaluate a novel mixture proposal distribution that allows us to robustly close large loops. We validate our approach experimentally for long camera trajectories processing thousands of images at reasonable frame rates

Details

ICRA Conference 2005 Conference Paper

Global A-Optimal Robot Exploration in SLAM

  • Robert Sim
  • Nicholas Roy

It is well-known that the Kalman filter for simultaneous localization and mapping (SLAM) converges to a fully correlated map in the limit of infinite time and data [1]. However, the rate of convergence of the map has a strong dependence on the order of the observations. We show that conventional exploration algorithms for collecting map data are sub-optimal in both the objective function and choice of optimization procedure. We show that optimizing the a-optimal information measure results in a more accurate map than existing approaches, using a greedy, closed-loop strategy. Secondly, we demonstrate that by restricting the planning to an appropriate policy class, we can tractably find non-greedy, global planning trajectories that produce more accurate maps, explicitly planning to close loops even in open-loop scenarios.

Details

IROS Conference 2005 Conference Paper

Stabilizing information-driven exploration for bearings-only SLAM using range gating

  • Robert Sim

This paper examines the problem of information-driven exploration for the purposes of simultaneous localization and mapping (SLAM) with a bearings-only sensor. In another work, we have demonstrated that employing an information-driven approach to exploration with an extended Kalman filter (EKF) can drive the robot to locations in the world where filter updates are ill-conditioned and linearization constraints are violated, potentially destabilizing the filter, and increasing the probability of divergence from the true state estimate. In this paper, we demonstrate an information-driven approach to exploration that preserves the stability of the EKF and produces maps that are significantly more accurate than a conventional information-driven approach. Our method is based on range-gating observations so as to avoid potentially destabilizing updates. We provide simulated experimental results demonstrating the superior performance of our approach over simple outlier gating and over heuristic-driven exploration.

Details

ICRA Conference 2005 Conference Paper

Stable Exploration for Bearings-only SLAM

  • Robert Sim

Recent work on robotic exploration and active sensing has examined a variety of information-theoretic approaches to efficient and convergent map construction. These involve moving an exploring robot to locations in the world where the anticipated information gain is maximized. In this paper we demonstrate that, for map construction using bearings-only information and the Extended Kalman Filter (EKF), driving exploration so as to maximize expected information gain leads to ill-conditioned filter updates and a high probability of divergence between the inferred map and reality. In particular, we present analytical and numerical results demonstrating the effects of blindly applying an information-theoretic approach to bearings-only exploration. Subsequently, we present experimental results demonstrating that an exploration approach that favours the conditioning of the filter update will lead to more accurate maps.

Details

IROS Conference 2004 Conference Paper

AQUA: an aquatic walking robot

  • Christina Georgiades
  • Andrew German
  • Andrew Hogue
  • Hui Liu
  • Chris Prahacs
  • Arlene Ripsman
  • Robert Sim
  • Luz Abril Torres-Méndez

This paper describes an underwater walking robotic system being developed under the name AQUA, the goals of the AQUA project, the overall hardware and software design, the basic hardware and sensor packages that have been developed, and some initial experiments. The robot is based on the RHex hexapod robot and uses a suite of sensing technologies, primarily based on computer vision and INS, to allow it to navigate and map clear shallow-water environments. The sensor-based navigation and mapping algorithms are based on the use of both artificial floating visual and acoustic landmarks as well as on naturally occurring underwater landmarks and trinocular stereo.

Details

IROS Conference 2004 Conference Paper

Landmark selection for vision-based navigation

  • Pablo Sala
  • Robert Sim
  • Ali Shokoufandeh
  • Sven J. Dickinson

Recent work in the object recognition community has yielded a class of interest point-based features that are stable under significant changes in scale, viewpoint, and illumination, making them ideally suited to landmark-based navigation. Although many such features may be visible in a given view of the robot's environment, only a few such features are necessary to estimate the robot's position and orientation. In this paper, we address the problem of automatically selecting, from the entire set of features visible in the robot's environment, the minimum (optimal) set by which the robot can navigate its environment. Specifically, we decompose the world into a small number of maximally sized regions such that at each position in a given region, the same small set of features is visible. We introduce a novel graph theoretic formulation of the problem and prove that it is NP-complete. Next, we introduce a number of approximation algorithms and evaluate them on both synthetic and real data.

Details

IROS Conference 2004 Conference Paper

Learning generative models of invariant features

  • Robert Sim
  • Gregory Dudek

We present a method for learning a set of models of visual features which are invariant to scale and translation in the image domain. The models are constructed by first applying the scale-invariant feature transform (SIFT) to a set of training images, and matching the extracted features across the images, followed by learning the pose-dependent behavior of the features. The modeling process avoids assumptions with respect to scene and imaging geometry, but rather learns the direct mapping from camera pose to feature observation. Such models are useful for applications to robotic tasks, such as localization, as well as visualization tasks. We present the model learning framework, and experimental results illustrating the success of the method for learning models that are useful for robot localization.

Details

ICRA Conference 2004 Conference Paper

Online Control Policy Optimization for Minimizing Map Uncertainty during Exploration

  • Robert Sim
  • Gregory Dudek
  • Nicholas Roy

Tremendous progress has been made recently in simultaneous localization and mapping of unknown environments. Using sensor and odometry data from an exploring mobile robot, it has become much easier to build high-quality globally consistent maps of many large, real-world environments. To date, however, relatively little attention has been paid to the controllers used to build these maps. Existing exploration strategies usually attempt to cover the largest amount of unknown space as quickly as possible. Few strategies exist for building the most reliable map possible, but the particular control strategy can have a substantial impact on the quality of the resulting map. In this paper, we devise a control algorithm for exploring unknown space that explicitly tries to build as large a map as possible while maintaining as accurate a map as possible. We make use of a parameterized class of spiral trajectory policies, choosing a new parameter setting at every time step to maximize the expected reward of the policy. We do this in the context of building a visual map of an unknown environment, and show that our strategy leads to a higher accuracy map faster than other candidate controllers, including any single choice in our policy class.

Details

AAAI Conference 2004 Conference Paper

Self-Organizing Visual Maps

  • Robert Sim
  • Gregory Dudek

This paper deals with automatically learning the spatial distribution of a set of images. That is, given a sequence of images acquired from well-separated locations, how can they be arranged to best explain their genesis? The solution to this problem can be viewed as an instance of robot mapping although it can also be used in other contexts. We examine the problem where only limited prior odometric information is available, employing a feature-based method derived from a probabilistic pose estimation framework. Initially, a set of visual features is selected from the images and correspondences are found across the ensemble. The images are then localized by first assembling the small subset of images for which odometric confidence is high, and sequentially inserting the remaining images, localizing each against the previous estimates, and taking advantage of any priors that are available. We present experimental results validating the approach, and demonstrating metrically and topologically accurate results over two large image ensembles. Finally, we discuss the results, their relationship to the autonomous exploration of an unknown environment, and their utility for robot localization and navigation.

PDF Details

IJCAI Conference 2003 Conference Paper

Comparing image-based localization methods

  • Robert Sim
  • Gregory Dudek

This paper compares alternative approaches to pose estimation using visual cues from the environment. We examine approaches that derive pose estimates from global image properties, such as principal components analysis (PCA) versus from local image properties, commonly referred to as landmarks. We also consider the failure-modes of the different methods. Our work is validated with experimental results.

PDF

IROS Conference 2003 Conference Paper

Effective exploration strategies for the construction of visual maps

  • Robert Sim
  • Gregory Dudek

We consider the effect of exploration policy in the context of the autonomous construction of a visual map of an unknown environment. Like other concurrent mapping and localization (CML) tasks, odometric uncertainty poses the problem of introducing distortions into the map which are difficult to correct without costly on-line or post-processing algorithms. Our problem is further compounded by the implicit nature of the visual map representation, which is designed to accommodate a wide variety of visual phenomena without assuming a particular imaging platform, thereby precluding the inference of scene geometry. Such a representation presents a requirement for a relatively dense sampling of observations of the environment in order to produce reliable models. Our goal is to develop an online policy for exploring an unknown environment which minimizes map distortion while maximizing coverage. We do not depend on costly post-hoc expectation maximization approaches to improve the output, but rather employ extended Kalman filter (EKF) methods to localize each observation once, and rely on the exploration policy to ensure that sufficient information is available to localize the successive observations. We present an experimental analysis of a variety of exploratory policies, in both simulated and real environments, and demonstrate that with an effective policy an accurate map can be constructed.

Details

ICRA Conference 2003 Conference Paper

Robodaemon -a device independent, network-oriented, modular mobile robot controller

  • Gregory Dudek
  • Robert Sim

We discuss a software environment for multi-robot, multi-platform mobile robot control and simulation. Like others, we have observed that mobile robotics research is greatly facilitated by the availability of a suitable simulator for both vehicle kinematics as well as sensing, and have created an environment that permits this while allowing a large measure of device independence. By using a multiprocessor internet-based architecture, our platform permits multiple users to use a variety of programming interfaces (visual, script-based or various application programming interfaces (API's)) to rapidly prototype methods to control multiple heterogeneous robots both in simulation and in real-world settings. We present an overview of our architecture and discuss its future directions.

Details

IROS Conference 2001 Conference Paper

Collaborative exploration for the construction of visual maps

  • Ioannis M. Rekleitis
  • Robert Sim
  • Gregory Dudek
  • Evangelos E. Milios

We examine the problem of learning a visual map of the environment while maintaining an accurate pose estimate. Our approach is based on using two robots in a simple collaborative scheme. Without outside information, as a robot collects training images, its position estimate accumulates errors, thus corrupting its knowledge of the positions from which observations are taken. We address this problem by deploying a second robot to observe the first one as it explores, thereby establishing a virtual tether, and enabling an accurate estimate of the robot's position while it constructs the map. We refer to this process as cooperative localization. The images collected during this process are assembled into a representation that allows vision-based position estimation from a single image at a later date. In addition to developing a formalism and concept, we validate our results experimentally and present quantitative results demonstrating the performance of the method in over 90 trials.

Details

ICRA Conference 1999 Conference Paper

Learning Visual Landmarks for Pose Estimation

  • Robert Sim
  • Gregory Dudek

We present an approach to vision-based mobile robot localization, even without an a-priori pose estimate. This is accomplished by learning a set of visual features called image-domain landmarks. The landmark learning mechanism is designed to be applicable to a wide range of environments. Each landmark is detected as a focal extremum of a measure of uniqueness and represented by an appearance-based encoding. Localization is performed using a method that matches observed landmarks to learned prototypes and generates independent position estimates for each match. The independent estimates are then combined to obtain a final position estimate, with an associated uncertainty. Quantitative experimental evidence is presented that demonstrates that accurate pose estimates can be obtained, despite changes to the environment.

Details

IROS Conference 1998 Conference Paper

Mobile robot localization from learned landmarks

  • Robert Sim
  • Gregory Dudek

Presents an approach to vision-based mobile robot localization. In an attempt to capitalize on the benefits of both image and landmark-based methods, we describe a method that combines their strengths. Images are encoded as a set of visual features called landmarks. Potential landmarks are detected using an attention mechanism implemented as a measure of uniqueness. They are then selected and represented by an appearance-based encoding. Localization is performed using a landmark tracking and interpolation method which obtains an estimate accurate to a fraction of the environment sampling density. Experimental results are shown to confirm the feasibility and accuracy of the method.

Details