Arrow Research search

Author name cluster

Heewoo Jun

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

2 papers
1 author row

Possible papers

2

ICML Conference 2020 Conference Paper

Distribution Augmentation for Generative Modeling

  • Heewoo Jun
  • Rewon Child
  • Mark Chen 0003
  • John Schulman
  • Aditya Ramesh
  • Alec Radford
  • Ilya Sutskever

We present distribution augmentation (DistAug), a simple and powerful method of regularizing generative models. Our approach applies augmentation functions to data and, importantly, conditions the generative model on the specific function used. Unlike typical data augmentation, DistAug allows usage of functions which modify the target density, enabling aggressive augmentations more commonly seen in supervised and self-supervised learning. We demonstrate this is a more effective regularizer than standard methods, and use it to train a 152M parameter autoregressive model on CIFAR-10 to 2. 56 bits per dim (relative to the state-of-the-art 2. 80). Samples from this model attain FID 12. 75 and IS 8. 40, outperforming the majority of GANs. We further demonstrate the technique is broadly applicable across model architectures and problem domains.

Details

ICML Conference 2020 Conference Paper

Generative Pretraining From Pixels

  • Mark Chen 0003
  • Alec Radford
  • Rewon Child
  • Jeffrey Wu 0003
  • Heewoo Jun
  • David Luan
  • Ilya Sutskever

Inspired by progress in unsupervised representation learning for natural language, we examine whether similar models can learn useful representations for images. We train a sequence Transformer to auto-regressively predict pixels, without incorporating knowledge of the 2D input structure. Despite training on low-resolution ImageNet without labels, we find that a GPT-2 scale model learns strong image representations as measured by linear probing, fine-tuning, and low-data classification. On CIFAR-10, we achieve 96. 3% accuracy with a linear probe, outperforming a supervised Wide ResNet, and 99. 0% accuracy with full fine-tuning, matching the top supervised pre-trained models. We are also competitive with self-supervised benchmarks on ImageNet when substituting pixels for a VQVAE encoding, achieving 69. 0% top-1 accuracy on a linear probe of our features.

Details