Author name cluster

Eun-Sol Kim

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

4 papers

1 author row

AAAI Conference 2024 Conference Paper

Structure-Aware Multimodal Sequential Learning for Visual Dialog

Young-Jin Kim
Min-Jun Kim
Kyunghwan An
Jinwoo Ahn
Jaeseok Kim
Yu-Jung Heo
Du-Seong Chang
Eun-Sol Kim

With the ability to collect vast amounts of image and natural language data from the web, there has been a remarkable advancement in Large-scale Language Models (LLMs). This progress has led to the emergence of chatbots and dialogue systems capable of fluent conversations with humans. As the variety of devices enabling interactions between humans and agents expands, and the performance of text-based dialogue systems improves, there has been recently proposed research on visual dialog. However, visual dialog requires understanding sequences of pairs consisting of images and sentences, making it challenging to gather sufficient data for training large-scale models from the web. In this paper, we propose a new multimodal learning method leveraging existing large-scale models designed for each modality, to enable model training for visual dialog with small visual dialog datasets. The key ideas of our approach are: 1) storing the history or context during the progression of visual dialog in the form of spatiotemporal graphs, and 2) introducing small modulation blocks between modality-specific models and the graphs to align the semantic spaces. For implementation, we introduce a novel structure-aware cross-attention method, which retrieves relevant image and text knowledge for utterance generation from the pretrained models. For experiments, we achieved a new state-of-the-art performance on three visual dialog datasets, including the most challenging one COMET.

PDF Details DOI

AAAI Conference 2021 Conference Paper

Image-to-Image Retrieval by Learning Similarity between Scene Graphs

Sangwoong Yoon
Woo Young Kang
Sungwook Jeon
SeongEun Lee
Changjin Han
Jonghun Park
Eun-Sol Kim

As a scene graph compactly summarizes the high-level content of an image in a structured and symbolic manner, the similarity between scene graphs of two images reflects the relevance of their contents. Based on this idea, we propose a novel approach for image-to-image retrieval using scene graph similarity measured by graph neural networks. In our approach, graph neural networks are trained to predict the proxy image relevance measure, computed from humanannotated captions using a pre-trained sentence similarity model. We collect and publish the dataset for image relevance measured by human annotators to evaluate retrieval algorithms. The collected dataset shows that our method agrees well with the human perception of image similarity than other competitive baselines.

PDF Details

AAAI Conference 2020 Conference Paper

Cut-Based Graph Learning Networks to Discover Compositional Structure of Sequential Video Data

Kyoung-Woon On
Eun-Sol Kim
Yu-Jung Heo
Byoung-Tak Zhang

Conventional sequential learning methods such as Recurrent Neural Networks (RNNs) focus on interactions between consecutive inputs, i. e. ﬁrst-order Markovian dependency. However, most of sequential data, as seen with videos, have complex dependency structures that imply variable-length semantic ﬂows and their compositions, and those are hard to be captured by conventional methods. Here, we propose Cut- Based Graph Learning Networks (CB-GLNs) for learning video data by discovering these complex structures of the video. The CB-GLNs represent video data as a graph, with nodes and edges corresponding to frames of the video and their dependencies respectively. The CB-GLNs ﬁnd compositional dependencies of the data in multilevel graph forms via a parameterized kernel with graph-cut and a message passing framework. We evaluate the proposed method on the two different tasks for video understanding: Video theme classiﬁcation (Youtube-8M dataset (Abu-El-Haija et al. 2016)) and Video Question and Answering (TVQA dataset (Lei et al. 2018)). The experimental results show that our model efﬁciently learns the semantic compositional structure of video data. Furthermore, our model achieves the highest performance in comparison to other baseline methods.

PDF Details

IJCAI Conference 2016 Conference Paper

DeepSchema: Automatic Schema Acquisition from Wearable Sensor Data in Restaurant Situations

Eun-Sol Kim
Kyoung-Woon On
Byoung-Tak Zhang

We explore the possibility of automatically constructing hierarchical schemas from low-level sensory data. Here we suggest a hierarchical event network to build the hierarchical schemas and describe a novel machine learning method to learn the network from the data. The traditional methods for describing schemas define the primitives and the relationships between them in advance. Therefore, it is difficult to adapt the constructed schemas in new situations. However, the proposed method constructs the schemas automatically from the data. Therefore, it has a novelty that the constructed schemas can be applied to new and unexpected situations flexibly. The key idea of constructing the hierarchical schema is selecting informative sensory data, integrating them sequentially and extracting high-level information. For the experiments, we collected sensory data using multiple wearable devices in restaurant situations. The experimental results demonstrate the real hierarchical schemas, which are probabilistic scripts and action primitives, constructed from the methods. Also, we show the constructed schemas can be used to predict the corresponding event to the low-level sensor data. Moreover, we show the prediction accuracy outperforms the conventional method significantly.

PDF Details