C. Lee Giles

NeSy Conference 2025 Conference Paper

Bridging Neural and Symbolic Computation: A Learnability Study of RNNs on Counter and Dyck Languages

• Neisarg Dave
• Daniel Kifer
• C. Lee Giles
• Ankur Mali

This work presents a neuro-symbolic analysis of the learnability of Recurrent Neural Networks (RNNs) in classifying structured formal languages—specifically, **counter languages** and **Dyck languages**, which serve as canonical examples of context-free and mildly context-sensitive grammars. While prior studies have highlighted the expressive power of first-order (LSTM) and second-order (O2RNN) architectures within the Chomsky hierarchy, we challenge this perspective by shifting the focus from theoretical expressivity to *practical learnability under finite precision constraints*. Our results suggest that RNNs function more as finite-state machines than stack-based automata when implemented with realistic training regimes and embedding representations. We show that classification performance degrades sharply as structural similarities between positive and negative sequences increase—highlighting a core limitation in the RNN’s ability to internalize hierarchical structure without symbolic scaffolding. Interestingly, even simple linear classifiers built on top of RNN-derived embeddings outperform chance, underscoring the hidden representational capacity within learned states. To probe generalization, we train models on input lengths up to 40 and evaluate on lengths extending to 500, using 10 distinct seeds to measure statistical robustness. O2RNNs consistently demonstrate greater stability and generalization compared to LSTMs, particularly under varied initialization strategies. These findings expose the fragility of learned language representations and emphasize the role of architectural bias, initialization, and data sampling in determining what is truly learnable. Ultimately, our study reframes RNN learnability through the lens of *symbolic structure and computational constraints*, advocating for stronger formal criteria when assessing neural models’ capacity to reason over structured sequences. We argue that expressivity alone is insufficient—**stability, precision, and symbolic alignment** are essential for true neuro-symbolic generalization.

AAMAS Conference 2023 Conference Paper

Artificial Prediction Markets Present a Novel Opportunity for Human-AI Collaboration

• Tatiana Chakravorti
• Vaibhav Singh
• Sarah Rajtmajer
• Michael McLaughlin
• Robert Fraleigh
• Christopher Griffin
• Anthony Kwasnica
• David Pennock

Despite high-profile successes in the field of Artificial Intelligence, machine-driven technologies still suffer important limitations, particularly for complex tasks where creativity, planning, common sense, intuition, or learning from limited data is required. These limitations motivate effective methods for human-machine collaboration. Our work makes two primary contributions. We thoroughly experiment with an artificial prediction market model to understand the effects of market parameters on model performance for benchmark classification tasks. We then demonstrate, through simulation, the impact of exogenous agents in the market, where these exogenous agents represent primitive human behaviors.

AAAI Conference 2023 Conference Paper

Backpropagation-Free Deep Learning with Recursive Local Representation Alignment

• Alexander G. Ororbia
• Ankur Mali
• Daniel Kifer
• C. Lee Giles

Training deep neural networks on large-scale datasets requires significant hardware resources whose costs (even on cloud platforms) put them out of reach of smaller organizations, groups, and individuals. Backpropagation (backprop), the workhorse for training these networks, is an inherently sequential process that is difficult to parallelize. Furthermore, researchers must continually develop various specialized techniques, such as particular weight initializations and enhanced activation functions, to ensure stable parameter optimization. Our goal is to seek an effective, neuro-biologically plausible alternative to backprop that can be used to train deep networks. In this paper, we propose a backprop-free procedure, recursive local representation alignment, for training large-scale architectures. Experiments with residual networks on CIFAR-10 and the large benchmark, ImageNet, show that our algorithm generalizes as well as backprop while converging sooner due to weight updates that are parallelizable and computationally less demanding. This is empirical evidence that a backprop-free algorithm can scale up to larger datasets.

AAAI Conference 2022 System Paper

A Synthetic Prediction Market for Estimating Confidence in Published Work

• Sarah Rajtmajer
• Christopher Griffin
• Jian Wu
• Robert Fraleigh
• Laxmaan Balaji
• Anna Squicciarini
• Anthony Kwasnica
• David Pennock

Explainably estimating confidence in published scholarly work offers opportunity for faster and more robust scientific progress. We develop a synthetic prediction market to assess the credibility of published claims in the social and behavioral sciences literature. We demonstrate our system and detail our findings using a collection of known replication projects. We suggest that this work lays the foundation for a research agenda that creatively uses AI for peer review.

AAAI Conference 2021 Conference Paper

Recognizing and Verifying Mathematical Equations using Multiplicative Differential Neural Units

• Ankur Mali
• Alexander G. Ororbia
• Daniel Kifer
• C. Lee Giles

Automated mathematical reasoning is a challenging problem that requires an agent to learn algebraic patterns that contain long-range dependencies. Two particular tasks that test this type of reasoning are (1) mathematical equation verification, which requires determining whether trigonometric and linear algebraic statements are valid identities or not, and (2) equation completion, which entails filling in a blank within an expression to make it true. Solving these tasks with deep learning requires that the neural model learn how to manipulate and compose various algebraic symbols, carrying this ability over to previously unseen expressions. Artificial neural networks, including recurrent networks and transformers, struggle to generalize on these kinds of difficult compositional problems, often exhibiting poor extrapolation performance. In contrast, recursive neural networks (recursive-NNs) are, theoretically, capable of achieving better extrapolation due to their tree-like design but are difficult to optimize as the depth of their underlying tree structure increases. To overcome this issue, we extend recursive-NNs to utilize multiplicative, higher-order synaptic connections and, furthermore, to learn to dynamically control and manipulate an external memory. We argue that this key modification gives the neural system the ability to capture powerful transition functions for each possible input. We demonstrate the effectiveness of our proposed higher-order, memory-augmented recursive-NN models on two challenging mathematical equation tasks, showing improved extrapolation, stable performance, and faster convergence. Our models achieve a 1. 53% average improvement over current state-of-the-art methods in equation verification and achieve a 2. 22% Top-1 average accuracy and 2. 96% Top- 5 average accuracy for equation completion.

AAAI Conference 2020 Conference Paper

Automatic Generation of Headlines for Online Math Questions

• Ke Yuan
• Dafang He
• Zhuoren Jiang
• Liangcai Gao
• Zhi Tang
• C. Lee Giles

Mathematical equations are an important part of dissemination and communication of scientific information. Students, however, often feel challenged in reading and understanding math content and equations. With the development of the Web, students are posting their math questions online. Nevertheless, constructing a concise math headline that gives a good description of the posted detailed math question is nontrivial. In this study, we explore a novel summarization task denoted as geNerating A concise Math hEadline from a detailed math question (NAME). Compared to conventional summarization tasks, this task has two extra and essential constraints: 1) Detailed math questions consist of text and math equations which require a unified framework to jointly model textual and mathematical information; 2) Unlike text, math equations contain semantic and structural features, and both of them should be captured together. To address these issues, we propose MathSum, a novel summarization model which utilizes a pointer mechanism combined with a multihead attention mechanism for mathematical representation augmentation. The pointer mechanism can either copy textual tokens or math tokens from source questions in order to generate math headlines. The multi-head attention mechanism is designed to enrich the representation of math equations by modeling and integrating both its semantic and structural features. For evaluation, we collect and make available two sets of real-world detailed math questions along with humanwritten math headlines, namely EXEQ-300k and OFEQ-10k. Experimental results demonstrate that our model (MathSum) significantly outperforms state-of-the-art models for both the EXEQ-300k and OFEQ-10k datasets.

AAAI Conference 2019 Conference Paper

Adversarial Training for Community Question Answer Selection Based on Multi-Scale Matching

• Xiao Yang
• Madian Khabsa
• Miaosen Wang
• Wei Wang
• Ahmed Hassan Awadallah
• Daniel Kifer
• C. Lee Giles

Community-based question answering (CQA) websites represent an important source of information. As a result, the problem of matching the most valuable answers to their corresponding questions has become an increasingly popular research topic. We frame this task as a binary (relevant/irrelevant) classification problem, and present an adversarial training framework to alleviate label imbalance issue. We employ a generative model to iteratively sample a subset of challenging negative samples to fool our classification model. Both models are alternatively optimized using REIN- FORCE algorithm. The proposed method is completely different from previous ones, where negative samples in training set are directly used or uniformly down-sampled. Further, we propose using Multi-scale Matching which explicitly inspects the correlation between words and ngrams of different levels of granularity. We evaluate the proposed method on SemEval 2016 and SemEval 2017 datasets and achieves state-of-the-art or similar performance.

AAAI Conference 2018 Conference Paper

Investigating Active Learning for Concept Prerequisite Learning

• Chen Liang
• Jianbo Ye
• Shuting Wang
• Bart Pursel
• C. Lee Giles

Concept prerequisite learning focuses on machine learning methods for measuring the prerequisite relation among concepts. With the importance of prerequisites for education, it has recently become a promising research direction. A major obstacle to extracting prerequisites at scale is the lack of large scale labels which will enable effective data driven solutions. We investigate the applicability of active learning to concept prerequisite learning. We propose a novel set of features tailored for prerequisite classification and compare the effectiveness of four widely used query strategies. Experimental results for domains including data mining, geometry, physics, and precalculus show that active learning can be used to reduce the amount of training data required. Given the proposed features, the query-by-committee strategy outperforms other compared query strategies.

IJCAI Conference 2017 Conference Paper

Learning to Read Irregular Text with Attention Mechanisms

• Xiao Yang
• Dafang He
• Zihan Zhou
• Daniel Kifer
• C. Lee Giles

We present a robust end-to-end neural-based model to attentively recognize text in natural images. Particularly, we focus on accurately identifying irregular (perspectively distorted or curved) text, which has not been well addressed in the previous literature. Previous research on text reading often works with regular (horizontal and frontal) text and does not adequately generalize to processing text with perspective distortion or curving effects. Our work proposes to overcome this difficulty by introducing two learning components: (1) an auxiliary dense character detection task that helps to learn text specific visual patterns, (2) an alignment loss that provides guidance to the training of an attention model. We show with experiments that these two components are crucial for achieving fast convergence and high classification accuracy for irregular text recognition. Our model outperforms previous work on two irregular-text datasets: SVT-Perspective and CUTE80, and is also highly-competitive on several regular-text datasets containing primarily horizontal and frontal text.

AAAI Conference 2012 Conference Paper

Name-Ethnicity Classification and Ethnicity-Sensitive Name Matching

• Pucktada Treeratpituk
• C. Lee Giles

Personal names are important and common information in many data sources, ranging from social networks and news articles to patient records and scientific documents. They are often used as queries for retrieving records and also as key information for linking documents from multiple sources. Matching personal names can be challenging due to variations in spelling and various formatting of names. While many approximated name matching techniques have been proposed, most are generic string-matching algorithms. Unlike other types of proper names, personal names are highly cultural. Many ethnicities have their own unique naming systems and identifiable characteristics. In this paper we explore such relationships between ethnicities and personal names to improve the name matching performance. First, we propose a name-ethnicity classifier based on the multinomial logistic regression. Our model can effectively identify nameethnicity from personal names in Wikipedia, which we use to define name-ethnicity, to within 85% accuracy. Next, we propose a novel alignment-based name matching algorithm, based on Smith–Waterman algorithm and logistic regression. Different name matching models are then trained for different name-ethnicity groups. Our preliminary experimental result on DBLP’s disambiguated author dataset yields a performance of 99% precision and 89% recall. Surprisingly, textual features carry more weight than phonetic ones in nameethnicity classification.

AAAI Conference 2012 Conference Paper

Table Header Detection and Classification

• Jing Fang
• Prasenjit Mitra
• Zhi Tang
• C. Lee Giles

In digital libraries, a table, as a specific document component as well as a condensed way to present structured and relational data, contains rich information and often the only source of. that information. In order to explore, retrieve, and reuse that data, tables should be identified and the data extracted. Table recognition is an old field of research. However, due to the diversity of table styles, the results are still far from satisfactory, and not a single algorithm performs well on all different types of tables. In this paper, we randomly take samples from the CiteSeerX to investigate diverse table styles for automatic table extraction. We find that table headers are one of the main characteristics of complex table styles. We identify a set of features that can be used to segregate headers from tabular data and build a classifier to detect table headers. Our empirical evaluation on PDF documents shows that using a Random Forest classifier achieves an accuracy of 92%.

IJCAI Conference 2011 Conference Paper

Context Sensitive Topic Models for Author Influence in Document Networks

• Saurabh Kataria
• Prasenjit Mitra
• Cornelia Caragea
• C. Lee Giles

Since the seminal work of Sampath et al. in 1996, despite the subsequent flourishing of techniques on diagnosis of discrete-event systems (DESs), the basic notions of fault and diagnosis have been remaining conceptually unchanged. Faults are defined at component level and diagnoses incorporate the occurrences of component faults within system evolutions: diagnosis is context-free. As this approach may be unsatisfactory for a complex DES, whose topology is organized in a hierarchy of abstractions, we propose to define different diagnosis rules for different subsystems in the hierarchy. Relevant fault patterns are specified as regular expressions on patterns of lower-level subsystems. Separation of concerns is achieved and the expressive power of diagnosis is enhanced: each subsystem has its proper set of diagnosis rules, which may or may not depend on the rules of other subsystems. Diagnosis is no longer anchored to components: it becomes context-sensitive. The approach yields seemingly contradictory but nonetheless possible scenarios: a subsystem can be normal despite the faulty behavior of a number of its components (positive paradox); also, it can be faulty despite the normal behavior of all its components (negative paradox).

IS Journal 2004 Journal Article

Collaborative Filtering with Maximum Entropy

• D. Pavlov
• E. Manavoglu
• D.M. Pennock
• C. Lee Giles

As users navigate through online document collections on high-volume Web servers, they depend on good recommendations. We present a novel maximum-entropy algorithm for generating accurate recommendations and a data-clustering approach for speeding up model training. Recommender systems attempt to automate the process of "word of mouth" recommendations within a community. Typical application environments such as online shops and search engines have many dynamic aspects.

AAMAS Conference 2003 Conference Paper

Don`t Move: The T-Rex Effect in the Predator-Prey World

• Todd M. Bacastow
• Brian R. Cook
• Kam-Chuen Jim
• C. Lee Giles

ICML Conference 2003 Conference Paper

Evolving Strategies for Focused Web Crawling

• Judy Johnson
• Kostas Tsioutsiouliklis
• C. Lee Giles

UAI Conference 2002 Conference Paper

Modelling Information Incorporation in Markets, with Application to Detecting and Explaining Events

• David M. Pennock
• Sandip Debnath
• Eric J. Glover
• C. Lee Giles

ICML Conference 2000 Conference Paper

A Normative Examination of Ensemble Learning Algorithms

• David M. Pennock
• Pedrito Maynard-Reid II
• C. Lee Giles
• Eric Horvitz

UAI Conference 2000 Conference Paper

Bayesian Classification and Feature Selection from Finite Data Sets

• Frans Coetzee
• Steve Lawrence
• C. Lee Giles

UAI Conference 2000 Conference Paper

Collaborative Filtering by Personality Diagnosis: A Hybrid Memory and Model-Based Approach

• David M. Pennock
• Eric Horvitz
• Steve Lawrence
• C. Lee Giles

The growth of Internet commerce has stimulated the use of collaborative filtering (CF) algorithms as recommender systems. Such systems leverage knowledge about the known preferences of multiple users to recommend items of interest to other users. CF methods have been harnessed to make recommendations about such items as web pages, movies, books, and toys. Researchers have proposed and evaluated many approaches for generating recommendations. We describe and evaluate a new method called emph{personality diagnosis (PD)}. Given a user's preferences for some items, we compute the probability that he or she is of the same ``personality type'' as other users, and, in turn, the probability that he or she will like new items. PD retains some of the advantages of traditional similarity-weighting techniques in that all data is brought to bear on each prediction and new data can be added easily and incrementally. Additionally, PD has a meaningful probabilistic interpretation, which may be leveraged to justify, explain, and augment results. We report empirical results on the EachMovie database of movie ratings, and on user profile data collected from the CiteSeer digital library of Computer Science research papers. The probabilistic framework naturally supports a variety of descriptive measurements---in particular, we consider the applicability of a value of information (VOI) computation.

ICML Conference 1997 Conference Paper

Predicting Multiprocessor Memory Access Patterns with Learning Models

• Majd F. Sakr
• Steven P. Levitan
• Donald M. Chiarulli
• Bill G. Horne
• C. Lee Giles

AAAI Conference 1997 Conference Paper

Presenting and Analyzing the Results of AI Experiments: Data Averaging and Data Snooping

• C. Lee Giles

Experimental resultsreportedin themachinelearning AI literaturecanbe misleading. This paperinvestigatesthe common processesof dataaveraging(reporting resultsin terms of the meanand standarddeviation of the resultsfrom multiple trials) and data snoopingin the context of neural networks, one of the most popular AI machine learning models. Both of theseprocesses canresult in misleading results and inaccurateconclusions. We demonstratehow easilythis canhappen and proposetechniquesfor avoiding thesevery important problems. For data averaging, common presentation assumesthat the distribution of individual results is Gaussian. However, we investigatethe distribution for common problems and find that it often does not approximate the Gaussiandistribution, may not be symmetric, and may be multimodal. We show that assumingGaussiandistributions cansignificantly affect the interpretation of results, especially thoseof comparisonstudies. For a controlled task, we find that the distribution of performance is skewedtowards better performance for smoothertarget functions and ’ skewedtowards worse performance for more complex target functions. We proposenew guidelines for reporting performance which provide more information about the actual distribution (e. g. box-whiskers plots). For data snooping, we demonstratethat optimization of performancevia experimentation with multiple parameterscanleadto significance being assignedto resultswhich aredue to chance. We suggestthat precisedescriptionsof experimentaltechniquescan be very important to the evaluation of results, and that we needto be awareof potential datasnoopingbiaseswhen formulating theseexperimental techniques (e. g. selecting the testprocedure). Additionally, it is important to only rely on appropriate statisticaltests and to ensurethat any assumptions madein the testsarevalid (e. g. normality of thedistribution).