Arrow Research search

Author name cluster

Yaser Sheikh

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

6 papers
2 author rows

Possible papers

6

NeurIPS Conference 2024 Conference Paper

Codec Avatar Studio: Paired Human Captures for Complete, Driveable, and Generalizable Avatars

  • Julieta Martinez
  • Emily Kim
  • Javier Romero
  • Timur Bagautdinov
  • Shunsuke Saito
  • Shoou-I Yu
  • Stuart Anderson
  • Michael Zollhöfer

To build photorealistic avatars that users can embody, human modelling must be complete (cover the full body), driveable (able to reproduce the current motion and appearance from the user), and generalizable ( i. e. , easily adaptable to novel identities). Towards these goals, paired captures, that is, captures of the same subject obtained from systems of diverse quality and availability, are crucial. However, paired captures are rarely available to researchers outside of dedicated industrial labs: Codec Avatar Studio is our proposal to close this gap. Towards generalization and driveability, we introduce a dataset of 256 subjects captured in two modalities: high resolution multi-view scans of their heads, and video from the internal cameras of a headset. Towards completeness, we introduce a dataset of 4 subjects captured in eight modalities: high quality relightable multi-view captures of heads and hands, full body multi-view captures with minimal and regular clothes, and corresponding head, hands and body phone captures. Together with our data, we also provide code and pre-trained models for different state-of-the-art human generation models. Our datasets and code are available at https: //github. com/facebookresearch/ava-256 and https: //github. com/facebookresearch/goliath.

PDF Details DOI

ICLR Conference 2021 Conference Paper

Neural Synthesis of Binaural Speech From Mono Audio

  • Alexander Richard
  • Dejan Markovic
  • Israel D. Gebru
  • Steven Krenn
  • Gladstone Alexander Butler
  • Fernando De la Torre
  • Yaser Sheikh

We present a neural rendering approach for binaural sound synthesis that can produce realistic and spatially accurate binaural sound in realtime. The network takes, as input, a single-channel audio source and synthesizes, as output, two-channel binaural sound, conditioned on the relative position and orientation of the listener with respect to the source. We investigate deficiencies of the l2-loss on raw waveforms in a theoretical analysis and introduce an improved loss that overcomes these limitations. In an empirical evaluation, we establish that our approach is the first to generate spatially accurate waveform outputs (as measured by real recordings) and outperforms existing approaches by a considerable margin, both quantitatively and in a perceptual study. Dataset and code are available online.

Details

NeurIPS Conference 2020 Conference Paper

Fully Convolutional Mesh Autoencoder using Efficient Spatially Varying Kernels

  • Yi Zhou
  • Chenglei Wu
  • Zimo Li
  • Chen Cao
  • Yuting Ye
  • Jason Saragih
  • Hao Li
  • Yaser Sheikh

Learning latent representations of registered meshes is useful for many 3D tasks. Techniques have recently shifted to neural mesh autoencoders. Although they demonstrate higher precision than traditional methods, they remain unable to capture fine-grained deformations. Furthermore, these methods can only be applied to a template-specific surface mesh, and is not applicable to more general meshes, like tetrahedrons and non-manifold meshes. While more general graph convolution methods can be employed, they lack performance in reconstruction precision and require higher memory usage. In this paper, we propose a non-template-specific fully convolutional mesh autoencoder for arbitrary registered mesh data. It is enabled by our novel convolution and (un)pooling operators learned with globally shared weights and locally varying coefficients which can efficiently capture the spatially varying contents presented by irregular mesh connections. Our model outperforms state-of-the-art methods on reconstruction accuracy. In addition, the latent codes of our network are fully localized thanks to the fully convolutional structure, and thus have much higher interpolation capability than many traditional 3D mesh generation models.

PDF Details

ICRA Conference 2016 Conference Paper

Real-time scalable 6DOF pose estimation for textureless objects

  • Zhe Cao 0003
  • Yaser Sheikh
  • Natasha Kholgade Banerjee

Real-time recognition of the 6DOF pose of textureless objects is a fundamental and challenging problem in robotics. We present a novel approach to perform real-time estimation of the viewpoint, scale, and translation of an object in RGB and RGB-D image captures. In this work, we use a 3D model to render example poses of a textureless object, and find the nearest match to the input image using a GPU implementation. To achieve invariance to illumination and appearance across an object, we transform images to the Laplacian of Gaussian space. To perform real-time matching, we introduce a novel reshaping of the template set and the image, and we restructure the traditional normalized cross-correlation operation to leverage the GPU for fast matrix-matrix multiplication. We provide futher speed up of large-scale template matching by contributing a dimensionality reduction approach using principal component analysis, and a candidate elimination method. Our method achieves state-of-the-art performance as shown by qualitative results and quantitative comparisons to pre-existing methods.

Details

NeurIPS Conference 2008 Conference Paper

Nonrigid Structure from Motion in Trajectory Space

  • Ijaz Akhter
  • Yaser Sheikh
  • Sohaib Khan
  • Takeo Kanade

Existing approaches to nonrigid structure from motion assume that the instantaneous 3D shape of a deforming object is a linear combination of basis shapes, which have to be estimated anew for each video sequence. In contrast, we propose that the evolving 3D structure be described by a linear combination of basis trajectories. The principal advantage of this lateral approach is that we do not need to estimate any basis vectors during computation. Instead, we show that generic bases over trajectories, such as the Discrete Cosine Transform (DCT) bases, can be used to effectively describe most real motions. This results in a significant reduction in unknowns, and corresponding stability, in estimation. We report empirical performance, quantitatively using motion capture data and qualitatively on several video sequences exhibiting nonrigid motions including piece-wise rigid motion, articulated motion, partially nonrigid motion (such as a facial expression), and highly nonrigid motion (such as a person dancing).

PDF Details
v2026.09.13