Author name cluster

Mark Weber

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

5 papers

2 author rows

NeurIPS Conference 2024 Conference Paper

An Image is Worth 32 Tokens for Reconstruction and Generation

Qihang Yu
Mark Weber
Xueqing Deng
Xiaohui Shen
Daniel Cremers
Liang-Chieh Chen

Recent advancements in generative models have highlighted the crucial role of image tokenization in the efficient synthesis of high-resolution images. Tokenization, which transforms images into latent representations, reduces computational demands compared to directly processing pixels and enhances the effectiveness and efficiency of the generation process. Prior methods, such as VQGAN, typically utilize 2D latent grids with fixed downsampling factors. However, these 2D tokenizations face challenges in managing the inherent redundancies present in images, where adjacent regions frequently display similarities. To overcome this issue, we introduce T ransformer-based 1-D i mensional Tok enizer (TiTok), an innovative approach that tokenizes images into 1D latent sequences. TiTok provides a more compact latent representation, yielding substantially more efficient and effective representations than conventional techniques. For example, a 256 × 256 × 3 image can be reduced to just 32 discrete tokens, a significant reduction from the 256 or 1024 tokens obtained by prior methods. Despite its compact nature, TiTok achieves competitive performance to state-of-the-art approaches. Specifically, using the same generator framework, TiTok attains 1. 97 gFID, outperforming MaskGIT baseline significantly by 4. 21 at ImageNet 256 × 256 benchmark. The advantages of TiTok become even more significant when it comes to higher resolution. At ImageNet 512 × 512 benchmark, TiTok not only outperforms state-of-the-art diffusion model DiT-XL/2 (gFID 2. 74 vs. 3. 04), but also reduces the image tokens by 64×, leading to 410× faster generation process. Our best-performing variant can significantly surpasses DiT-XL/2 (gFID 2. 13 vs. 3. 04) while still generating high-quality samples 74× faster. Codes and models are available at https: //github. com/bytedance/1d-tokenizer

PDF Details DOI

TMLR Journal 2024 Journal Article

MaskBit: Embedding-free Image Generation via Bit Tokens

Mark Weber
Lijun Yu
Qihang Yu
Xueqing Deng
Xiaohui Shen
Daniel Cremers
Liang-Chieh Chen

Masked transformer models for class-conditional image generation have become a compelling alternative to diffusion models. Typically comprising two stages - an initial VQGAN model for transitioning between latent space and image space, and a subsequent Transformer model for image generation within latent space - these frameworks offer promising avenues for image synthesis. In this study, we present two primary contributions: Firstly, an empirical and systematic examination of VQGANs, leading to a modernized VQGAN. Secondly, a novel embedding-free generation network operating directly on bit tokens - a binary quantized representation of tokens with rich semantics. The first contribution furnishes a transparent, reproducible, and high-performing VQGAN model, enhancing accessibility and matching the performance of current state-of-the-art methods while revealing previously undisclosed details. The second contribution demonstrates that embedding-free image generation using bit tokens achieves a new state-of-the-art FID of 1.52 on the ImageNet $256\times256$ benchmark, with a compact generator model of mere 305M parameters. The code for this project is available on https://github.com/markweberdev/maskbit.

PDF Details

NeurIPS Conference 2021 Conference Paper

STEP: Segmenting and Tracking Every Pixel

Mark Weber
Jun Xie
Yukun Zhu
Paul Voigtlaender
Bo Chen
Bradley Green
Andreas Geiger
Bastian Leibe

The task of assigning semantic classes and track identities to every pixel in a video is called video panoptic segmentation. Our work is the first that targets this task in a real-world setting requiring dense interpretation in both spatial and temporal domains. As the ground-truth for this task is difficult and expensive to obtain, existing datasets are either constructed synthetically or only sparsely annotated within short video clips. To overcome this, we introduce a new benchmark encompassing two datasets, KITTI-STEP, and MOTChallenge-STEP. The datasets contain long video sequences, providing challenging examples and a test-bed for studying long-term pixel-precise segmentation and tracking under real-world conditions. We further propose a novel evaluation metric Segmentation and Tracking Quality (STQ) that fairly balances semantic and tracking aspects of this task and is more appropriate for evaluating sequences of arbitrary length. Finally, we provide several baselines to evaluate the status of existing methods on this new challenging dataset. We have made our datasets, metric, benchmark servers, and baselines publicly available, and hope this will inspire future research.

PDF Details

ICRA Conference 2020 Conference Paper

4D Generic Video Object Proposals

Aljosa Osep
Paul Voigtlaender
Mark Weber
Jonathon Luiten
Bastian Leibe

Many high-level video understanding methods require input in the form of object proposals. Currently, such proposals are predominantly generated with the help of neural networks that were trained for detecting and segmenting a set of known object classes, which limits their applicability to cases where all objects of interest are represented in the training set. We propose an approach that can reliably extract spatio-temporal object proposals for both known and unknown object categories from stereo video. Our 4D Generic Video Tubes (4D-GVT) method combines motion cues, stereo data, and data-driven object instance segmentation in a probabilistic framework to compute a compact set of video-object proposals that precisely localizes object candidates and their contours in 3D space and time.

Details

IROS Conference 2020 Conference Paper

Single-Shot Panoptic Segmentation

Mark Weber
Jonathon Luiten
Bastian Leibe

We present a novel end-to-end single-shot method that segments countable object instances (things) as well as background regions (stuff) into a non-overlapping panoptic segmentation at almost video frame rate. Current state-of-the-art methods are far from reaching video frame rate and mostly rely on merging instance segmentation with semantic background segmentation, making them impractical to use in many applications such as robotics. Our approach relaxes this requirement by using an object detector but is still able to re-solve inter- and intra-class overlaps to achieve a non-overlapping segmentation. On top of a shared encoder-decoder backbone, we utilize multiple branches for semantic segmentation, object detection, and instance center prediction. Finally, our panoptic head combines all outputs into a panoptic segmentation and can even handle conflicting predictions between branches as well as certain false predictions. Our network achieves 32. 6% PQ on MS-COCO at 23. 5 FPS, opening up panoptic segmentation to a broader field of applications.

Details