Aaron Hertzmann

NeurIPS Conference 2025 Conference Paper

Fast Data Attribution for Text-to-Image Models

• Sheng-Yu Wang
• Aaron Hertzmann
• Alexei Efros
• Richard Zhang
• Jun-Yan Zhu

Data attribution for text-to-image models aims to identify the training images that most significantly influenced a generated output. Existing attribution methods involve considerable computational resources for each query, making them impractical for real-world applications. We propose a novel approach for scalable and efficient data attribution. Our key idea is to distill a slow, unlearning-based attribution method to a feature embedding space for efficient retrieval of highly influential training images. During deployment, combined with efficient indexing and search methods, our method successfully finds highly influential images without running expensive attribution algorithms. We show extensive results on both medium-scale models trained on MSCOCO and large-scale Stable Diffusion models trained on LAION, demonstrating that our method can achieve better or competitive performance in a few seconds, faster than existing methods by 2, 500x - 400, 000x. Our work represents a meaningful step towards the large-scale application of data attribution methods on real-world models such as Stable Diffusion.

NeurIPS Conference 2024 Conference Paper

Data Attribution for Text-to-Image Models by Unlearning Synthesized Images

• Sheng-Yu Wang
• Aaron Hertzmann
• Alexei A. Efros
• Jun-Yan Zhu
• Richard Zhang

The goal of data attribution for text-to-image models is to identify the training images that most influence the generation of a new image. Influence is defined such that, for a given output, if a model is retrained from scratch without the most influential images, the model would fail to reproduce the same output. Unfortunately, directly searching for these influential images is computationally infeasible, since it would require repeatedly retraining models from scratch. In our work, we propose an efficient data attribution method by simulating unlearning the synthesized image. We achieve this by increasing the training loss on the output image, without catastrophic forgetting of other, unrelated concepts. We then identify training images with significant loss deviations after the unlearning process and label these as influential. We evaluate our method with a computationally intensive but "gold-standard" retraining from scratch and demonstrate our method's advantages over previous methods.

NeurIPS Conference 2020 Conference Paper

GANSpace: Discovering Interpretable GAN Controls

• Erik Härkönen
• Aaron Hertzmann
• Jaakko Lehtinen
• Sylvain Paris

This paper describes a simple technique to analyze Generative Adversarial Networks (GANs) and create interpretable controls for image synthesis, such as change of viewpoint, aging, lighting, and time of day. We identify important latent directions based on Principal Component Analysis (PCA) applied either in latent space or feature space. Then, we show that a large number of interpretable controls can be defined by layer-wise perturbation along the principal directions. Moreover, we show that BigGAN can be controlled with layer-wise inputs in a StyleGAN-like manner. We show results on different GANs trained on various datasets, and demonstrate good qualitative matches to edit directions found through earlier supervised approaches.

NeurIPS Conference 2013 Conference Paper

Efficient Optimization for Sparse Gaussian Process Regression

• Yanshuai Cao
• Marcus Brubaker
• David Fleet
• Aaron Hertzmann

We propose an efficient discrete optimization algorithm for selecting a subset of training data to induce sparsity for Gaussian process regression. The algorithm estimates this inducing set and the hyperparameters using a single objective, either the marginal likelihood or a variational free energy. The space and time complexity are linear in the training set size, and the algorithm can be applied to large regression problems on discrete or continuous domains. Empirical evaluation shows state-of-art performance in the discrete case and competitive results in the continuous case.

IROS Conference 2009 Conference Paper

Prioritized optimization for task-space control

• Martin de Lasa
• Aaron Hertzmann

We introduce an optimization framework called prioritized optimization control, in which a nested sequence of objectives are optimized so as not to conflict with higher-priority objectives. We focus on the case of quadratic objectives and derive an efficient recursive solver for this case. We show how task-space control can be formulated in this framework, and demonstrate the technique on three sample control problems. The proposed formulation supports acceleration, torque, and bilateral force constraints, while simplifying reasoning about task-space control. This scheme unifies prioritized task-space and optimization-based control. Our method computes control torques for all presented examples in real-time.

ICML Conference 2007 Conference Paper

Multifactor Gaussian process models for style-content separation

• Jack M. Wang
• David J. Fleet
• Aaron Hertzmann

NeurIPS Conference 2005 Conference Paper

Gaussian Process Dynamical Models

• Jack Wang
• Aaron Hertzmann
• David Fleet

This paper introduces Gaussian Process Dynamical Models (GPDM) for nonlinear time series analysis. A GPDM comprises a low-dimensional latent space with associated dynamics, and a map from the latent space to an observation space. We marginalize out the model parameters in closed-form, using Gaussian Process (GP) priors for both the dynamics and the observation mappings. This results in a nonparametric model for dynamical systems that accounts for uncertainty in the model. We demonstrate the approach on human motion capture data in which each pose is 62-dimensional. Despite the use of small data sets, the GPDM learns an effective representation of the nonlinear dynamics in these spaces. Webpage: http: //www. dgp. toronto. edu/

NeurIPS Conference 2005 Conference Paper

Learning Shared Latent Structure for Image Synthesis and Robotic Imitation

• Aaron Shon
• Keith Grochow
• Aaron Hertzmann
• Rajesh Rao

We propose an algorithm that uses Gaussian process regression to learn common hidden structure shared between corresponding sets of heterogenous observations. The observation spaces are linked via a single, reduced-dimensionality latent variable space. We present results from two datasets demonstrating the algorithms's ability to synthesize novel data from learned correspondences. We first show that the method can learn the nonlinear mapping between corresponding views of objects, filling in missing data as needed to synthesize novel views. We then show that the method can learn a mapping between human degrees of freedom and robotic degrees of freedom for a humanoid robot, allowing robotic imitation of human poses from motion capture data.

NeurIPS Conference 2003 Conference Paper

Learning Non-Rigid 3D Shape from 2D Motion

• Lorenzo Torresani
• Aaron Hertzmann
• Christoph Bregler

This paper presents an algorithm for learning the time-varying shape of a non-rigid 3D object from uncalibrated 2D tracking data. We model shape motion as a rigid component (rotation and translation) combined with a non-rigid deformation. Reconstruction is ill-posed if arbitrary deforma- tions are allowed. We constrain the problem by assuming that the object shape at each time instant is drawn from a Gaussian distribution. Based on this assumption, the algorithm simultaneously estimates 3D shape and motion for each time frame, learns the parameters of the Gaussian, and robustly fills-in missing data points. We then extend the algorithm to model temporal smoothness in object shape, thus allowing it to handle severe cases of missing data.