Sudipta Sengupta

ICML Conference 2024 Conference Paper

Bifurcated Attention for Single-Context Large-Batch Sampling

• Ben Athiwaratkun
• Sujan Kumar Gonugondla
• Sanjay Krishna Gouda
• Haifeng Qian
• Hantian Ding
• Qing Sun 0013
• Jun Wang 0022
• Jiacheng Guo

In our study, we present bifurcated attention, a method developed for language model inference in single-context batch sampling contexts. This approach aims to reduce redundant memory IO costs, a significant factor in latency for high batch sizes and long context lengths. Bifurcated attention achieves this by dividing the attention mechanism during incremental decoding into two distinct GEMM operations, focusing on the KV cache from prefill and the decoding process. This method ensures precise computation and maintains the usual computational load (FLOPs) of standard attention mechanisms, but with reduced memory IO. Bifurcated attention is also compatible with multi-query attention mechanism known for reduced memory IO for KV cache, further enabling higher batch size and context length. The resulting efficiency leads to lower latency, improving suitability for real-time applications, e. g. , enabling massively-parallel answer generation without substantially increasing latency, enhancing performance when integrated with post-processing techniques such as reranking.

NeurIPS Conference 2024 Conference Paper

Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scale

• Tianyue Ou
• Frank F. Xu
• Aman Madaan
• Jiarui Liu
• Robert Lo
• Abishek Sridhar
• Sudipta Sengupta
• Dan Roth

LLMs can now act as autonomous agents that interact with digital environments and complete specific objectives (e. g. , arranging an online meeting). However, accuracy is still far from satisfactory, partly due to a lack of large-scale, direct demonstrations for digital tasks. Obtaining supervised data from humans is costly, and automatic data collection through exploration or reinforcement learning relies on complex environmental and content setup, resulting in datasets that lack comprehensive coverage of various scenarios. On the other hand, there is abundant knowledge that may indirectly assist task completion, such as online tutorials that were created for human consumption. In this work, we present Synatra, an approach that effectively transforms this indirect knowledge into direct supervision at scale. We define different types of indirect knowledge, and carefully study the available sources to obtain it, methods to encode the structure of direct demonstrations, and finally methods to transform indirect knowledge into direct demonstrations. We use 100k such synthetically-created demonstrations to finetune a 7B CodeLlama, and demonstrate that the resulting agent surpasses all comparably sized models on three web-based task benchmarks Mind2Web, MiniWoB++ and WebArena, as well as surpassing GPT-3. 5 on WebArena and Mind2Web. In addition, while synthetic demonstrations prove to be only 3% the cost of human demonstrations (at $0. 031 each), we show that the synthetic demonstrations can be more effective than an identical number of human demonstrations collected from limited domains.

ICML Conference 2022 Conference Paper

Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval

• Uri Alon 0002
• Frank F. Xu
• Junxian He
• Sudipta Sengupta
• Dan Roth 0001
• Graham Neubig

Retrieval-based language models (R-LM) model the probability of natural language text by combining a standard language model (LM) with examples retrieved from an external datastore at test time. While effective, a major bottleneck of using these models in practice is the computationally costly datastore search, which can be performed as frequently as every time step. In this paper, we present RetoMaton - retrieval automaton - which approximates the datastore search, based on (1) saving pointers between consecutive datastore entries, and (2) clustering of entries into "states". This effectively results in a weighted finite automaton built on top of the datastore, instead of representing the datastore as a flat list. The creation of the automaton is unsupervised, and a RetoMaton can be constructed from any text collection: either the original training corpus or from another domain. Traversing this automaton at inference time, in parallel to the LM inference, reduces its perplexity by up to 1. 85, or alternatively saves up to 83% of the nearest neighbor searches over $k$NN-LM (Khandelwal et al. , 2020) without hurting perplexity. Our code and trained models are available at https: //github. com/neulab/retomaton.

STOC Conference 2000 Conference Paper

epsilon-optimization schemes and L-bit precision: alternative perspectives in combinatorial optimization (extended abstract)

• James B. Orlin
• Andreas S. Schulz
• Sudipta Sengupta