Author name cluster

Yusuke Kato

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

5 papers

2 author rows

NeurIPS Conference 2025 Conference Paper

LaViDa: A Large Diffusion Language Model for Multimodal Understanding

Shufan Li
Konstantinos Kallidromitis
Hritik Bansal
Akash Gokul
Yusuke Kato
Kazuki Kozuka
Jason Kuen
Zhe Lin

Modern Vision-Language Models (VLMs) can solve a wide range of tasks requiring visual reasoning. In real-world scenarios, desirable properties for VLMs include fast inference and controllable generation (e. g. , constraining outputs to adhere to a desired format). However, existing autoregressive (AR) VLMs like LLaVA struggle in these aspects. Discrete diffusion models (DMs) offer a promising alternative, enabling parallel decoding for faster inference and bidirectional context for controllable generation through text-infilling. While effective in language-only settings, DMs' potential for multimodal tasks is underexplored. We introduce LaViDa, a family of VLMs built on DMs. We build LaViDa by equipping DMs with a vision encoder and jointly fine-tune the combined parts for multimodal instruction following. To address challenges encountered, LaViDa incorporates novel techniques such as complementary masking for effective training, prefix KV cache for efficient inference, and timestep shifting for high-quality sampling. Experiments show that LaViDa achieves competitive or superior performance to AR VLMs on multi-modal benchmarks such as MMMU, while offering unique advantages of DMs, including flexible speed-quality tradeoff, controllability, and bidirectional reasoning. On COCO captioning, LaViDa surpasses Open-LLaVa-Next-8B by +4. 1 CIDEr with 1. 92x speedup. On bidirectional tasks, it achieves +59% improvement on Constrained Poem Completion. These results demonstrate LaViDa as a strong alternative to AR VLMs. Code and models is available at https: //github. com/jacklishufan/LaViDa

PDF Details

ICLR Conference 2025 Conference Paper

SegLLM: Multi-round Reasoning Segmentation with Large Language Models

Xudong Wang 0007
Shaolun Zhang
Shufan Li
Kehan Li
Konstantinos Kallidromitis
Yusuke Kato
Kazuki Kozuka
Trevor Darrell

We present SegLLM, a novel multi-round interactive reasoning segmentation model that enhances LLM-based segmentation by exploiting conversational memory of both visual and textual outputs. By leveraging a mask-aware multimodal LLM, SegLLM re-integrates previous segmentation results into its input stream, enabling it to reason about complex user intentions and segment objects in relation to previously identified entities, including positional, interactional, and hierarchical relationships, across multiple interactions. This capability allows SegLLM to respond to visual and text queries in a chat-like manner. Evaluated on the newly curated MRSeg benchmark, SegLLM outperforms existing methods in multi- round interactive reasoning segmentation by over 20%. Additionally, we observed that training on multi-round reasoning segmentation data enhances performance on standard single-round referring segmentation and localization tasks, resulting in a 5.5% increase in cIoU for referring expression segmentation and a 4.5% improvement in Acc@0.5 for referring expression localization.

Details

NeurIPS Conference 2024 Conference Paper

Aligning Diffusion Models by Optimizing Human Utility

Shufan Li
Konstantinos Kallidromitis
Akash Gokul
Yusuke Kato
Kazuki Kozuka

We present Diffusion-KTO, a novel approach for aligning text-to-image diffusion models by formulating the alignment objective as the maximization of expected human utility. Unlike previous methods, Diffusion-KTO does not require collecting pairwise preference data nor training a complex reward model. Instead, our objective uses per-image binary feedback signals, e. g. likes or dislikes, to align the model with human preferences. After fine-tuning using Diffusion-KTO, text-to-image diffusion models exhibit improved performance compared to existing techniques, including supervised fine-tuning and Diffusion-DPO, both in terms of human judgment and automatic evaluation metrics such as PickScore and ImageReward. Overall, Diffusion-KTO unlocks the potential of leveraging readily available per-image binary preference signals and broadens the applicability of aligning text-to-image diffusion models with human preferences.

PDF Details DOI

NeurIPS Conference 2023 Conference Paper

Hierarchical Open-vocabulary Universal Image Segmentation

Xudong Wang
Shufan Li
Konstantinos Kallidromitis
Yusuke Kato
Kazuki Kozuka
Trevor Darrell

Open-vocabulary image segmentation aims to partition an image into semantic regions according to arbitrary text descriptions. However, complex visual scenes can be naturally decomposed into simpler parts and abstracted at multiple lev4 els of granularity, introducing inherent segmentation ambiguity. Unlike existing methods that typically sidestep this ambiguity and treat it as an external factor, our approach actively incorporates a hierarchical representation encompassing different semantic-levels into the learning process. We propose a decoupled text-image fusion mechanism and representation learning modules for both “things” and “stuff”. Additionally, we systematically examine the differences that exist in the textual and visual features between these types of categories. Our resulting model, named HIPIE, tackles HIerarchical, oPen-vocabulary, and unIvErsal segmentation tasks within a unified framework. Benchmarked on diverse datasets, e. g. , ADE20K, COCO, Pascal-VOC Part, and RefCOCO/RefCOCOg, HIPIE achieves the state-of14 the-art results at various levels of image comprehension, including semantic-level (e. g. , semantic segmentation), instance-level (e. g. , panoptic/referring segmentationand object detection), as well as part-level (e. g. , part/subpart segmentation) tasks.

PDF Details

IROS Conference 2019 Conference Paper

Adjusting Weight of Action Decision in Exploration for Logistics Warehouse Picking Learning

Yusuke Kato
Tomoaki Nakamura
Takayuki Nagai
Natsuki Yamanobe
Kazuyuki Nagata
Jun Ozawa

The purpose of this study is for a robot to learn picking motions in a logistics warehouse environment. The picking operation performed by a robot often fails owing to the inclination of items placed on a shelf, as well as the minimum clearance between the products and their vinyl packaging. Therefore, we considered acquiring a specific motion trajectory by reinforcement learning. However, because numerous types of items are handled in logistics warehouses, efficient learning is required. Therefore, in this research, we propose a method to efficiently exploration for learning picking an object by determining a focus exploration area for learning based on previous results of different objects.

Details