Nan Ding 0002

ICLR Conference 2024 Conference Paper

CausalLM is not optimal for in-context learning

• Nan Ding 0002
• Tomer Levinboim
• Jialin Wu
• Sebastian Goodman
• Radu Soricut

Recent empirical evidence indicates that transformer based in-context learning performs better when using a prefix language model (prefixLM), in which in-context samples can all attend to each other, compared to causal language models (causalLM), which use auto-regressive attention that prohibits in-context samples to attend to future samples. While this result is intuitive, it is not understood from a theoretical perspective. In this paper we take a theoretical approach and analyze the convergence behavior of prefixLM and causalLM under a certain parameter construction. Our analysis shows that both LM types converge to their stationary points at a linear rate, but that while prefixLM converges to the optimal solution of linear regression, causalLM convergence dynamics follows that of an online gradient descent algorithm, which is not guaranteed to be optimal even as the number of samples grows infinitely. We supplement our theoretical claims with empirical experiments over synthetic and real tasks and using various types of transformers. Our experiments verify that causalLM consistently underperforms prefixLM in all settings.

ICLR Conference 2023 Conference Paper

PaLI: A Jointly-Scaled Multilingual Language-Image Model

• Xi Chen 0071
• Xiao Wang 0038
• Soravit Changpinyo
• A. J. Piergiovanni
• Piotr Padlewski
• Daniel Salz
• Sebastian Goodman
• Adam Grycner

Effective scaling and a flexible task interface enable large language models to excel at many tasks. We present PaLI, a model that extends this approach to the joint modeling of language and vision. PaLI generates text based on visual and textual inputs, and with this interface performs many vision, language, and multimodal tasks, in many languages. To train PaLI, we make use of large pretrained encoder-decoder language models and Vision Transformers (ViTs). This allows us to capitalize on their existing capabilities and leverage the substantial cost of training them. We find that joint scaling of the vision and language components is important. Since existing Transformers for language are much larger than their vision counterparts, we train a large, 4-billion parameter ViT (ViT-e) to quantify the benefits from even larger-capacity vision models. To train PaLI, we create a large multilingual mix of pretraining tasks, based on a new image-text training set containing 10B images and texts in over 100 languages. PaLI achieves state-of-the-art in multiple vision and language tasks (such as captioning, visual question-answering, scene-text understanding), while retaining a simple, modular, and scalable design.

ICML Conference 2012 Conference Paper

Dependent Hierarchical Normalized Random Measures for Dynamic Topic Modeling

• Changyou Chen
• Nan Ding 0002
• Wray L. Buntine

ICML Conference 2012 Conference Paper

Robust Classification with Adiabatic Quantum Optimization

• Vasil S. Denchev
• Nan Ding 0002
• S. V. N. Vishwanathan
• Hartmut Neven