Author name cluster

Yu-Jung Heo

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

4 papers

1 author row

AAAI Conference 2024 Conference Paper

Structure-Aware Multimodal Sequential Learning for Visual Dialog

Young-Jin Kim
Min-Jun Kim
Kyunghwan An
Jinwoo Ahn
Jaeseok Kim
Yu-Jung Heo
Du-Seong Chang
Eun-Sol Kim

With the ability to collect vast amounts of image and natural language data from the web, there has been a remarkable advancement in Large-scale Language Models (LLMs). This progress has led to the emergence of chatbots and dialogue systems capable of fluent conversations with humans. As the variety of devices enabling interactions between humans and agents expands, and the performance of text-based dialogue systems improves, there has been recently proposed research on visual dialog. However, visual dialog requires understanding sequences of pairs consisting of images and sentences, making it challenging to gather sufficient data for training large-scale models from the web. In this paper, we propose a new multimodal learning method leveraging existing large-scale models designed for each modality, to enable model training for visual dialog with small visual dialog datasets. The key ideas of our approach are: 1) storing the history or context during the progression of visual dialog in the form of spatiotemporal graphs, and 2) introducing small modulation blocks between modality-specific models and the graphs to align the semantic spaces. For implementation, we introduce a novel structure-aware cross-attention method, which retrieves relevant image and text knowledge for utterance generation from the pretrained models. For experiments, we achieved a new state-of-the-art performance on three visual dialog datasets, including the most challenging one COMET.

PDF Details DOI

AAAI Conference 2021 Conference Paper

DramaQA: Character-Centered Video Story Understanding with Hierarchical QA

Seongho Choi
Kyoung-Woon On
Yu-Jung Heo
Ahjeong Seo
Youwon Jang
Minsu Lee
Byoung-Tak Zhang

Despite recent progress on computer vision and natural language processing, developing a machine that can understand video story is still hard to achieve due to the intrinsic difficulty of video story. Moreover, researches on how to evaluate the degree of video understanding based on human cognitive process have not progressed as yet. In this paper, we propose a novel video question answering (Video QA) task, DramaQA, for a comprehensive understanding of the video story. The DramaQA focuses on two perspectives: 1) Hierarchical QAs as an evaluation metric based on the cognitive developmental stages of human intelligence. 2) Character-centered video annotations to model local coherence of the story. Our dataset is built upon the TV drama “Another Miss Oh”1 and it contains 17, 983 QA pairs from 23, 928 various length video clips, with each QA pair belonging to one of four difficulty levels. We provide 217, 308 annotated images with rich charactercentered annotations, including visual bounding boxes, behaviors and emotions of main characters, and coreference resolved scripts. Additionally, we suggest Multi-level Context Matching model which hierarchically understands charactercentered representations of video to answer questions. We release our dataset and model publicly for research purposes2, and we expect our work to provide a new perspective on video story understanding research.

PDF Details

AAAI Conference 2020 Conference Paper

Cut-Based Graph Learning Networks to Discover Compositional Structure of Sequential Video Data

Kyoung-Woon On
Eun-Sol Kim
Yu-Jung Heo
Byoung-Tak Zhang

Conventional sequential learning methods such as Recurrent Neural Networks (RNNs) focus on interactions between consecutive inputs, i. e. ﬁrst-order Markovian dependency. However, most of sequential data, as seen with videos, have complex dependency structures that imply variable-length semantic ﬂows and their compositions, and those are hard to be captured by conventional methods. Here, we propose Cut- Based Graph Learning Networks (CB-GLNs) for learning video data by discovering these complex structures of the video. The CB-GLNs represent video data as a graph, with nodes and edges corresponding to frames of the video and their dependencies respectively. The CB-GLNs ﬁnd compositional dependencies of the data in multilevel graph forms via a parameterized kernel with graph-cut and a message passing framework. We evaluate the proposed method on the two different tasks for video understanding: Video theme classiﬁcation (Youtube-8M dataset (Abu-El-Haija et al. 2016)) and Video Question and Answering (TVQA dataset (Lei et al. 2018)). The experimental results show that our model efﬁciently learns the semantic compositional structure of video data. Furthermore, our model achieves the highest performance in comparison to other baseline methods.

PDF Details

NeurIPS Conference 2018 Conference Paper

Answerer in Questioner's Mind: Information Theoretic Approach to Goal-Oriented Visual Dialog

Sang-Woo Lee
Yu-Jung Heo
Byoung-Tak Zhang

Goal-oriented dialog has been given attention due to its numerous applications in artificial intelligence. Goal-oriented dialogue tasks occur when a questioner asks an action-oriented question and an answerer responds with the intent of letting the questioner know a correct action to take. To ask the adequate question, deep learning and reinforcement learning have been recently applied. However, these approaches struggle to find a competent recurrent neural questioner, owing to the complexity of learning a series of sentences. Motivated by theory of mind, we propose "Answerer in Questioner's Mind" (AQM), a novel information theoretic algorithm for goal-oriented dialog. With AQM, a questioner asks and infers based on an approximated probabilistic model of the answerer. The questioner figures out the answerer’s intention via selecting a plausible question by explicitly calculating the information gain of the candidate intentions and possible answers to each question. We test our framework on two goal-oriented visual dialog tasks: "MNIST Counting Dialog" and "GuessWhat? !". In our experiments, AQM outperforms comparative algorithms by a large margin.

PDF Details