Raymond J. Mooney

AAAI Conference 2021 Conference Paper

Deep Just-In-Time Inconsistency Detection Between Comments and Source Code

• Sheena Panthaplackel
• Junyi Jessy Li
• Milos Gligoric
• Raymond J. Mooney

Natural language comments convey key aspects of source code such as implementation, usage, and pre- and postconditions. Failure to update comments accordingly when the corresponding code is modified introduces inconsistencies, which is known to lead to confusion and software bugs. In this paper, we aim to detect whether a comment becomes inconsistent as a result of changes to the corresponding body of code, in order to catch potential inconsistencies just-in-time, i. e. , before they are committed to a code base. To achieve this, we develop a deep-learning approach that learns to correlate a comment with code changes. By evaluating on a large corpus of comment/code pairs spanning various comment types, we show that our model outperforms multiple baselines by significant margins. For extrinsic evaluation, we show the usefulness of our approach by combining it with a comment update model to build a more comprehensive automatic comment maintenance system which can both detect and resolve inconsistent comments based on code changes.

AAAI Conference 2021 Conference Paper

Dialog Policy Learning for Joint Clarification and Active Learning Queries

• Aishwarya Padmakumar
• Raymond J. Mooney

Intelligent systems need to be able to recover from mistakes, resolve uncertainty, and adapt to novel concepts not seen during training. Dialog interaction can enable this by the use of clarifications for correction and resolving uncertainty, and active learning queries to learn new concepts encountered during operation. Prior work on dialog systems has either focused on exclusively learning how to perform clarification/ information seeking, or to perform active learning. In this work, we train a hierarchical dialog policy to jointly perform both clarification and active learning in the context of an interactive language-based image retrieval task motivated by an online shopping application, and demonstrate that jointly learning dialog policies for clarification and active learning is more effective than the use of static dialog policies for one or both of these functions.

AAAI Conference 2020 Conference Paper

Associating Natural Language Comment and Source Code Entities

• Sheena Panthaplackel
• Milos Gligoric
• Raymond J. Mooney
• Junyi Jessy Li

Comments are an integral part of software development; they are natural language descriptions associated with source code elements. Understanding explicit associations can be useful in improving code comprehensibility and maintaining the consistency between code and comments. As an initial step towards this larger goal, we address the task of associating entities in Javadoc comments with elements in Java source code. We propose an approach for automatically extracting supervised data using revision histories of open source projects and present a manually annotated evaluation dataset for this task. We develop a binary classifier and a sequence labeling model by crafting a rich feature set which encompasses various aspects of code, comments, and the relationships between them. Experiments show that our systems outperform several baselines learning from the proposed supervision.

JAIR Journal 2020 Journal Article

Jointly Improving Parsing and Perception for Natural Language Commands through Human-Robot Dialog

• Jesse Thomason
• Aishwarya Padmakumar
• Jivko Sinapov
• Nick Walker
• Yuqian Jiang
• Harel Yedidsion
• Justin Hart
• Peter Stone

In this work, we present methods for using human-robot dialog to improve language understanding for a mobile robot agent. The agent parses natural language to underlying semantic meanings and uses robotic sensors to create multi-modal models of perceptual concepts like red and heavy. The agent can be used for showing navigation routes, delivering objects to people, and relocating objects from one location to another. We use dialog clari_cation questions both to understand commands and to generate additional parsing training data. The agent employs opportunistic active learning to select questions about how words relate to objects, improving its understanding of perceptual concepts. We evaluated this agent on Amazon Mechanical Turk. After training on data induced from conversations, the agent reduced the number of dialog questions it asked while receiving higher usability ratings. Additionally, we demonstrated the agent on a robotic platform, where it learned new perceptual concepts on the y while completing a real-world task.

IJCAI Conference 2019 Conference Paper

Using Natural Language for Reward Shaping in Reinforcement Learning

• Prasoon Goyal
• Scott Niekum
• Raymond J. Mooney

Recent reinforcement learning (RL) approaches have shown strong performance in complex domains, such as Atari games, but are highly sample inefficient. A common approach to reduce interaction time with the environment is to use reward shaping, which involves carefully designing reward functions that provide the agent intermediate rewards for progress towards the goal. Designing such rewards remains a challenge, though. In this work, we use natural language instructions to perform reward shaping. We propose a framework that maps free-form natural language instructions to intermediate rewards, that can seamlessly be integrated into any standard reinforcement learning algorithm. We experiment with Montezuma's Revenge from the Atari video games domain, a popular benchmark in RL. Our experiments on a diverse set of 15 tasks demonstrate that for the same number of interactions with the environment, using language-based rewards can successfully complete the task 60% more often, averaged across all tasks, compared to learning without language.

IJCAI Conference 2017 Conference Paper

Multi-Modal Word Synset Induction

• Jesse Thomason
• Raymond J. Mooney

A word in natural language can be polysemous, having multiple meanings, as well as synonymous, meaning the same thing as other words. Word sense induction attempts to find the senses of polysemous words. Synonymy detection attempts to find when two words are interchangeable. We combine these tasks, first inducing word senses and then detecting similar senses to form word-sense synonym sets (synsets) in an unsupervised fashion. Given pairs of images and text with noun phrase labels, we perform synset induction to produce collections of underlying concepts described by one or more noun phrases. We find that considering multi-modal features from both visual and textual context yields better induced synsets than using either context alone. Human evaluations show that our unsupervised, multi-modally induced synsets are comparable in quality to annotation-assisted ImageNet synsets, achieving about 84% of ImageNet synsets' approval.

IJCAI Conference 2017 Conference Paper

Stacking With Auxiliary Features

• Nazneen Fatema Rajani
• Raymond J. Mooney

Ensembling methods are well known for improving prediction accuracy. However, they are limited in the sense that they cannot effectively discriminate among component models. In this paper, we propose stacking with auxiliary features that learns to fuse additional relevant information from multiple component systems as well as input instances to improve performance. We use two types of auxiliary features -- instance features and provenance features. The instance features enable the stacker to discriminate across input instances and the provenance features enable the stacker to discriminate across component systems. When combined together, our algorithm learns to rely on systems that not just agree on an output but also the provenance of this output in conjunction with the properties of the input instance. We demonstrate the success of our approach on three very different and challenging natural language and vision problems: Slot Filling, Entity Discovery and Linking, and ImageNet Object Detection. We obtain new state-of-the-art results on the first two tasks and significant improvements on the ImageNet task, thus verifying the power and generality of our approach.

IJCAI Conference 2016 Conference Paper

Learning Multi-Modal Grounded Linguistic Semantics by Playing "I Spy"

• Jesse Thomason
• Jivko Sinapov
• Maxwell Svetlik
• Peter Stone
• Raymond J. Mooney

Grounded language learning bridges words like red and square with robot perception. The vast majority of existing work in this space limits robot perception to vision. In this paper, we build perceptual models that use haptic, auditory, and proprioceptive data acquired through robot exploratory behaviors to go beyond vision. Our system learns to ground natural language words describing objects using supervision from an interactive human-robot I Spy game. In this game, the human and robot take turns describing one object among several, then trying to guess which object the other has described. All supervision labels were gathered from human participants physically present to play this game with a robot. We demonstrate that our multi-modal system for grounding natural language outperforms a traditional, vision-only grounding framework by comparing the two on the "I Spy" task. We also provide a qualitative analysis of the groundings learned in the game, visualizing what words are understood better with multi-modal sensory information as well as identifying learned word meanings that correlate with physical object properties (e. g. "small" negatively correlates with object weight).

IJCAI Conference 2009 Conference Paper

• Lilyana Mihalkova
• Raymond J. Mooney

A central goal of transfer learning is to enable learning when training data from the domain of interest is limited. Yet, work on transfer across relational domains has so far focused on the case where there is a significant amount of target data. This paper bridges this gap by studying transfer when the amount of target data is minimal and consists of information about just a handful of entities. In the extreme case, only a single entity is known. We present the SR2LR algorithm that finds an effective mapping of predicates from a source model to the target domain in this setting and thus renders preexisting knowledge useful to the target task. We demonstrate SR2LR’s effectiveness in three benchmark relational domains on social interactions and study its behavior as information about an increasing number of entities becomes available.

AAAI Conference 2008 Conference Paper

Learning to Connect Language and Perception

• Raymond J. Mooney

To truly understand language, an intelligent system must be able to connect words, phrases, and sentences to its perception of objects and events in the world. Current natural language processing and computer vision systems make extensive use of machine learning to acquire the probabilistic knowledge needed to comprehend linguistic and visual input. However, to date, there has been relatively little work on learning the relationships between the two modalities. In this talk, I will review some of the existing work on learning to connect language and perception, discuss important directions for future research in this area, and argue that the time is now ripe to make a concerted effort to address this important, integrative AI problem.

AIIM Journal 2005 Journal Article

Comparative experiments on learning information extractors for proteins and their interactions

• Razvan Bunescu
• Ruifang Ge
• Rohit J. Kate
• Edward M. Marcotte
• Raymond J. Mooney
• Arun K. Ramani
• Yuk Wah Wong

Objective: Automatically extracting information from biomedical text holds the promise of easily consolidating large amounts of biological knowledge in computer-accessible form. This strategy is particularly attractive for extracting data relevant to genes of the human genome from the 11 million abstracts in Medline. However, extraction efforts have been frustrated by the lack of conventions for describing human genes and proteins. We have developed and evaluated a variety of learned information extraction systems for identifying human protein names in Medline abstracts and subsequently extracting information on interactions between the proteins. Methods and Material: We used a variety of machine learning methods to automatically develop information extraction systems for extracting information on gene/protein name, function and interactions from Medline abstracts. We present cross-validated results on identifying human proteins and their interactions by training and testing on a set of approximately 1000 manually-annotated Medline abstracts that discuss human genes/proteins. Results: We demonstrate that machine learning approaches using support vector machines and maximum entropy are able to identify human proteins with higher accuracy than several previous approaches. We also demonstrate that various rule induction methods are able to identify protein interactions with higher precision than manually-developed rules. Conclusion: Our results show that it is promising to use machine learning to automatically build systems for extracting information from biomedical text. The results also give a broad picture of the relative strengths of a wide variety of methods when tested on a reasonably large human-annotated corpus.

JMLR Journal 2003 Journal Article

Bottom-Up Relational Learning of Pattern Matching Rules for Information Extraction

• Mary Elaine Califf
• Raymond J. Mooney

Information extraction is a form of shallow text processing that locates a specified set of relevant items in a natural-language document. Systems for this task require significant domain-specific knowledge and are time-consuming and difficult to build by hand, making them a good application for machine learning. We present an algorithm, RAPIER, that uses pairs of sample documents and filled templates to induce pattern-match rules that directly extract fillers for the slots in the template. RAPIER is a bottom-up learning algorithm that incorporates techniques from several inductive logic programming systems. We have implemented the algorithm in a system that allows patterns to have constraints on the words, part-of-speech tags, and semantic classes present in the filler and the surrounding text. We present encouraging experimental results on two domains. [abs] [ pdf ][ ps.gz ][ ps ]

IJCAI Conference 2003 Conference Paper

Constructing Diverse Classifier Ensembles using Artificial Training Examples

• Prem Melville
• Raymond J. Mooney

Ensemble methods like bagging and boosting that combine the decisions of multiple hypotheses are some of the strongest existing machine learning methods. The diversity of the members of an ensemble is known to be an important factor in determining its generalization error. This paper presents a new method for generating ensembles that directly constructs diverse hypotheses using additional artificially-constructed training examples. The technique is a simple, general metalearner that can use any strong learner as a base classifier to build diverse committees. Experimental results using decision-tree induction as a base learner demonstrate that this approach consistently achieves higher predictive accuracy than both the base classifier and bagging (whereas boosting can occasionally decrease accuracy), and also obtains higher accuracy than boosting early in the learning curve when training data is limited.

AAAI Conference 1999 Conference Paper

Automatic Construction of Semantic Lexicons for Learning Natural Language Interfaces

• Cynthia A. Thompson
• Stanford University
• Raymond J. Mooney
• University of Texas

This paper describes a system, Wolfie (WOrd Learning From Interpreted Examples), that acquires a semantic lexicon from a corpus of sentences paired with semantic representations. The lexicon learned consists of words paired with meaning representations. Wolfie is part of an integrated system that learns to parse novel sentences into semantic representations, such as logical database queries. Experimental results are presented demonstrating Wolfie’s ability to learn useful lexicons for a database interface in four different natural languages. The lexicons learned by Wolfie are compared to those acquired by a similar system developed by Siskind (1996).

AAAI Conference 1999 Conference Paper

Relational Learning of Pattern-Match Rules for Information Extraction

• Mary Elaine Califf
• Illinois State University
• Raymond J. Mooney
• University of Texas at Austin

Information extraction is a form of shallow text processingthat locates a specified set of relevant items in a naturaManguage document. Systems for this task require significant domain-specific knowledgeand are time-consumingand difficult to build by hand, makingthem a good application for machinelearning. Wepresent a system, RAPIER, that uses pairs of sample documentsand filled templates to inducepattern-matchrules that directly extract fillers for the slots in the template. RAPIER employs a bottom-up learning algorithm whichincorporates techniques fromseveral inductive logic programming systems and acquires unboundedpatterns that include constraints on the words, part-of-speech tags, and semantic classes present in the filler andthe surroundingtext. We present encouraging experimental results on two domains.

IJCAI Conference 1997 Conference Paper

Learning to Improve both Efficiency and Quality of Planning

• Tara A. Estlin
• Raymond J. Mooney

Most research in learning for planning has concentrated on efficiency gains. Another important goal is improving the quality of final plans. Learning to improve plan quality has been examined by a few researchers, however, little research has been done learning to improve both efficiency and quality. This paper explores this problem by using the SCOPE learning system to acquire control knowledge that improves on both of these metrics. Since SCOPE uses a very flexible training approach, we can easily focus its learning algorithm to prefer search paths that are better for particular evaluation metrics. Experimental results show that SCOPE can significantly improve both the quality of final plans and overall planning efficiency.

AIJ Journal 1994 Journal Article

Theory refinement combining analytical and empirical methods

• Dirk Ourston
• Raymond J. Mooney

This article describes a comprehensive system for automatic theory (knowledge base) refinement. The system applies to classification tasks employing a propositional Horn-clause domain theory. Given an imperfect domain theory and a set of training examples, the approach uses partial and incorrect proofs to identify potentially faulty rules. For each faulty rule, subsets of examples are used to inductively generate a correction. Because the system starts with an approximate domain theory, fewer training examples are generally required to attain a given level of classification accuracy compared to a purely empirical learning system. The system has been tested in two previously explored application domains: recognizing important classes of DNA sequences and diagnosing diseased soybean plants.

AAAI Conference 1986 Conference Paper

A Domain Independent Explanation-Based Generalizer

• Raymond J. Mooney

A domain independent technique for generalizing a broad class of explanations is described. This method is compared and contrasted with other approaches to generalizing explanations, including an abstract version of the algorithm used in the STRIPS system and the EBG technique recently developed by Mitchell, Keller, and Kedar-Cabelli. We have tested this generalization technique on a number examples in different domains, and present detailed descriptions of several of these.