Author name cluster

Ajay Jain

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

10 papers

2 author rows

ICLR Conference 2023 Conference Paper

DreamFusion: Text-to-3D using 2D Diffusion

Ben Poole
Ajay Jain
Jonathan T. Barron
Ben Mildenhall

Recent breakthroughs in text-to-image synthesis have been driven by diffusion models trained on billions of image-text pairs. Adapting this approach to 3D synthesis would require large-scale datasets of labeled 3D or multiview data and efficient architectures for denoising 3D data, neither of which currently exist. In this work, we circumvent these limitations by using a pretrained 2D text-to-image diffusion model to perform text-to-3D synthesis. We introduce a loss based on probability density distillation that enables the use of a 2D diffusion model as a prior for optimization of a parametric image generator. Using this loss in a DeepDream-like procedure, we optimize a randomly-initialized 3D model (a Neural Radiance Field, or NeRF) via gradient descent such that its 2D renderings from random angles achieve a low loss. The resulting 3D model of the given text can be viewed from any angle, relit by arbitrary illumination, or composited into any 3D environment. Our approach requires no 3D training data and no modifications to the image diffusion model, demonstrating the effectiveness of pretrained image diffusion models as priors.

Details

NeurIPS Conference 2023 Conference Paper

Video Prediction Models as Rewards for Reinforcement Learning

Alejandro Escontrela
Ademi Adeniji
Wilson Yan
Ajay Jain
Xue Bin Peng
Ken Goldberg
Youngwoon Lee
Danijar Hafner

Specifying reward signals that allow agents to learn complex behaviors is a long-standing challenge in reinforcement learning. A promising approach is to extract preferences for behaviors from unlabeled videos, which are widely available on the internet. We present Video Prediction Rewards (VIPER), an algorithm that leverages pretrained video prediction models as action-free reward signals for reinforcement learning. Specifically, we first train an autoregressive transformer on expert videos and then use the video prediction likelihoods as reward signals for a reinforcement learning agent. VIPER enables expert-level control without programmatic task rewards across a wide range of DMC, Atari, and RLBench tasks. Moreover, generalization of the video prediction model allows us to derive rewards for an out-of-distribution environment where no expert data is available, enabling cross-embodiment generalization for tabletop manipulation. We see our work as starting point for scalable reward specification from unlabeled videos that will benefit from the rapid advances in generative modeling. Source code and datasets are available on the project website: https: //ViperRL. com

PDF Details

UAI Conference 2022 Conference Paper

AdaCat: Adaptive categorical discretization for autoregressive models

Qiyang Li
Ajay Jain
Pieter Abbeel

Autoregressive generative models can estimate complex continuous data distributions, like trajectory rollouts in an RL environment, image intensities, and audio. Most state-of-the-art models discretize continuous data into several bins and use categorical distributions over the bins to approximate the continuous data distribution. The advantage is that the categorical distribution can easily express multiple modes and are straightforward to optimize. However, such approximation cannot express sharp changes in density without using significantly more bins, which makes it parameter inefficient. We propose an efficient, expressive, multimodal parameterization called Adaptive Categorical Discretization (AdaCat). AdaCat discretizes each dimension of an autoregressive model adaptively, which allows the model to allocate density to fine intervals of interest, improving parameter efficiency. AdaCat generalizes both categoricals and quantile-based regression. AdaCat is a simple add-on to any discretization-based distribution estimator. In experiments, AdaCat improves density estimation for real-world tabular data, images, audio, and trajectories, and improves planning in model-based offline RL.

Details

NeurIPS Conference 2020 Conference Paper

Denoising Diffusion Probabilistic Models

Jonathan Ho
Ajay Jain
Pieter Abbeel

We present high quality image synthesis results using diffusion probabilistic models, a class of latent variable models inspired by considerations from nonequilibrium thermodynamics. Our best results are obtained by training on a weighted variational bound designed according to a novel connection between diffusion probabilistic models and denoising score matching with Langevin dynamics, and our models naturally admit a progressive lossy decompression scheme that can be interpreted as a generalization of autoregressive decoding. On the unconditional CIFAR10 dataset, we obtain an Inception score of 9. 46 and a state-of-the-art FID score of 3. 17. On 256x256 LSUN, we obtain sample quality similar to ProgressiveGAN.

PDF Details

UAI Conference 2020 Conference Paper

Locally Masked Convolution for Autoregressive Models

Ajay Jain
Pieter Abbeel
Deepak Pathak

High-dimensional generative models have many applications including image compression, multimedia generation, anomaly detection and data completion. State-of-the-art estimators for natural images are autoregressive, decomposing the joint distribution over pixels into a product of conditionals parameterized by a deep neural network, e. g. a convolutional neural network such as the PixelCNN. However, PixelCNNs only model a single decomposition of the joint, and only a single generation order is efficient. For tasks such as image completion, these models are unable to use much of the observed context. To generate data in arbitrary orders, we introduce LMConv: a simple modification to the standard 2D convolution that allows arbitrary masks to be applied to the weights at each location in the image. Using LMConv, we learn an ensemble of distribution estimators that share parameters but differ in generation order, achieving improved performance on whole-image density estimation (2. 89 bpd on unconditional CIFAR10), as well as globally coherent image completions. Code is available at https: //ajayjain. github. io/lmconv.

Details

NeurIPS Conference 2020 Conference Paper

Sparse Graphical Memory for Robust Planning

Scott Emmons
Ajay Jain
Misha Laskin
Thanard Kurutach
Pieter Abbeel
Deepak Pathak

To operate effectively in the real world, agents should be able to act from high-dimensional raw sensory input such as images and achieve diverse goals across long time-horizons. Current deep reinforcement and imitation learning methods can learn directly from high-dimensional inputs but do not scale well to long-horizon tasks. In contrast, classical graphical methods like A* search are able to solve long-horizon tasks, but assume that the state space is abstracted away from raw sensory input. Recent works have attempted to combine the strengths of deep learning and classical planning; however, dominant methods in this domain are still quite brittle and scale poorly with the size of the environment. We introduce Sparse Graphical Memory (SGM), a new data structure that stores states and feasible transitions in a sparse memory. SGM aggregates states according to a novel two-way consistency objective, adapting classic state aggregation criteria to goal-conditioned RL: two states are redundant when they are interchangeable both as goals and as starting states. Theoretically, we prove that merging nodes according to two-way consistency leads to an increase in shortest path lengths that scales only linearly with the merging threshold. Experimentally, we show that SGM significantly outperforms current state of the art methods on long horizon, sparse-reward visual navigation tasks. Project video and code are available at https: //sites. google. com/view/sparse-graphical-memory.

PDF Details

NeurIPS Conference 1993 Conference Paper

A Comparison of Dynamic Reposing and Tangent Distance for Drug Activity Prediction

Thomas Dietterich
Ajay Jain
Richard Lathrop
Tomás Lozano-Pérez

In drug activity prediction (as in handwritten character recogni(cid: 173) tion), the features extracted to describe a training example depend on the pose (location, orientation, etc. ) of the example. In hand(cid: 173) written character recognition, one of the best techniques for ad(cid: 173) dressing this problem is the tangent distance method of Simard, LeCun and Denker (1993). Jain, et al. (1993a; 1993b) introduce a new technique-dynamic reposing-that also addresses this prob(cid: 173) lem. Dynamic reposing iteratively learns a neural network and then reposes the examples in an effort to maximize the predicted out(cid: 173) put values. New models are trained and new poses computed until models and poses converge. This paper compares dynamic reposing to the tangent distance method on the task of predicting the bio(cid: 173) logical activity of musk compounds. In a 20-fold cross-validation,

PDF Details

NeurIPS Conference 1991 Conference Paper

Generalization Performance in PARSEC - A Structured Connectionist Parsing Architecture

Ajay Jain

This paper presents PARSEC-a system for generating connectionist parsing networks from example parses. PARSEC is not based on formal grammar systems and is geared toward spoken language tasks. PARSEC networks exhibit three strengths important for application to speech pro(cid: 173) cessing: 1) they learn to parse, and generalize well compared to hand(cid: 173) coded grammars; 2) they tolerate several types of noise; 3) they can learn to use multi-modal input. Presented are the PARSEC architecture and performance analyses along several dimensions that demonstrate PARSEC's features. PARSEC's performance is compared to that of tra(cid: 173) ditional grammar-based parsing systems.

PDF Details

NeurIPS Conference 1991 Conference Paper

JANUS: Speech-to-Speech Translation Using Connectionist and Non-Connectionist Techniques

Alex Waibel
Ajay Jain
Arthur McNair
Joe Tebelskis
Louise Osterholtz
Hiroaki Saito
Otto Schmidbauer
Tilo Sloboda

We present JANUS, a speech-to-speech translation system that utilizes diverse processing strategies, including connectionist learning, tradi(cid: 173) tional AI knowledge representation approaches, dynamic programming, and stochastic techniques. JANUS translates continuously spoken English and German into German, English, and Japanese. JANUS cur(cid: 173) rently achieves 87% translation fidelity from English speech and 97% from German speech. We present the JANUS system along with com(cid: 173) parative evaluations of its interchangeable processing components, with special emphasis on the connectionist modules. • Also with University of Karlsruhe, Karlsruhe. Germany. 1N"ow with Alliant Techsystems Research and Technology Center. Hopkins. Minnesota.

PDF Details

NeurIPS Conference 1989 Conference Paper

Incremental Parsing by Modular Recurrent Connectionist Networks

Ajay Jain
Alex Waibel

We present a novel, modular, recurrent connectionist network architec(cid: 173) ture which learns to robustly perform incremental parsing of complex sentences. From sequential input, one word at a time, our networks learn to do semantic role assignment, noun phrase attachment, and clause structure recognition for sentences with passive constructions and center embedded clauses. The networks make syntactic and semantic predictions at every point in time, and previous predictions are revised as expectations are affirmed or violated with the arrival of new informa(cid: 173) tion. Our networks induce their own "grammar rules" for dynamically transforming an input sequence of words into a syntactic/semantic in(cid: 173) terpretation. These networks generalize and display tolerance to input which has been corrupted in ways common in spoken language.

PDF Details