Author name cluster

Michael Ryoo

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

7 papers

1 author row

NeurIPS Conference 2022 Conference Paper

Does Self-supervised Learning Really Improve Reinforcement Learning from Pixels?

Xiang Li
Jinghuan Shang
Srijan Das
Michael Ryoo

We investigate whether self-supervised learning (SSL) can improve online reinforcement learning (RL) from pixels. We extend the contrastive reinforcement learning framework (e. g. , CURL) that jointly optimizes SSL and RL losses and conduct an extensive amount of experiments with various self-supervised losses. Our observations suggest that the existing SSL framework for RL fails to bring meaningful improvement over the baselines only taking advantage of image augmentation when the same amount of data and augmentation is used. We further perform evolutionary searches to find the optimal combination of multiple self-supervised losses for RL, but find that even such a loss combination fails to meaningfully outperform the methods that only utilize carefully designed image augmentations. After evaluating these approaches together in multiple different environments including a real-world robot environment, we confirm that no single self-supervised loss or image augmentation method can dominate all environments and that the current framework for joint optimization of SSL and RL is limited. Finally, we conduct the ablation study on multiple factors and demonstrate the properties of representations learned with different approaches.

PDF Details

NeurIPS Conference 2022 Conference Paper

Learning Viewpoint-Agnostic Visual Representations by Recovering Tokens in 3D Space

Jinghuan Shang
Srijan Das
Michael Ryoo

Humans are remarkably flexible in understanding viewpoint changes due to visual cortex supporting the perception of 3D structure. In contrast, most of the computer vision models that learn visual representation from a pool of 2D images often fail to generalize over novel camera viewpoints. Recently, the vision architectures have shifted towards convolution-free architectures, visual Transformers, which operate on tokens derived from image patches. However, these Transformers do not perform explicit operations to learn viewpoint-agnostic representation for visual understanding. To this end, we propose a 3D Token Representation Layer (3DTRL) that estimates the 3D positional information of the visual tokens and leverages it for learning viewpoint-agnostic representations. The key elements of 3DTRL include a pseudo-depth estimator and a learned camera matrix to impose geometric transformations on the tokens, trained in an unsupervised fashion. These enable 3DTRL to recover the 3D positional information of the tokens from 2D patches. In practice, 3DTRL is easily plugged-in into a Transformer. Our experiments demonstrate the effectiveness of 3DTRL in many vision tasks including image classification, multi-view video alignment, and action recognition. The models with 3DTRL outperform their backbone Transformers in all the tasks with minimal added computation. Our code is available at https: //github. com/elicassion/3DTRL.

PDF Details

NeurIPS Conference 2021 Conference Paper

TokenLearner: Adaptive Space-Time Tokenization for Videos

Michael Ryoo
AJ Piergiovanni
Anurag Arnab
Mostafa Dehghani
Anelia Angelova

In this paper, we introduce a novel visual representation learning which relies on a handful of adaptively learned tokens, and which is applicable to both image and video understanding tasks. Instead of relying on hand-designed splitting strategies to obtain visual tokens and processing a large number of densely sampled patches for attention, our approach learns to mine important tokens in visual data. This results in efficiently and effectively finding a few important visual tokens and enables modeling of pairwise attention between such tokens, over a longer temporal horizon for videos, or the spatial content in image frames. Our experiments demonstrate strong performance on several challenging benchmarks for video recognition tasks. Importantly, due to our tokens being adaptive, we accomplish competitive results at significantly reduced computational cost. We establish new state-of-the-arts on multiple video datasets, including Kinetics-400, Kinetics-600, Charades, and AViD.

PDF Details

NeurIPS Conference 2020 Conference Paper

AViD Dataset: Anonymized Videos from Diverse Countries

AJ Piergiovanni
Michael Ryoo

We introduce a new public video dataset for action recognition: Anonymized Videos from Diverse countries (AViD). Unlike existing public video datasets, AViD is a collection of action videos from many different countries. The motivation is to create a public dataset that would benefit training and pretraining of action recognition models for everybody, rather than making it useful for limited countries. Further, all the face identities in the AViD videos are properly anonymized to protect their privacy. It also is a static dataset where each video is licensed with the creative commons license. We confirm that most of the existing video datasets are statistically biased to only capture action videos from a limited number of countries. We experimentally illustrate that models trained with such biased datasets do not transfer perfectly to action videos from the other countries, and show that AViD addresses such problem. We also confirm that the new AViD dataset could serve as a good dataset for pretraining the models, performing comparably or better than prior datasets. The dataset is available at https: //github. com/piergiaj/AViD

PDF Details

AAAI Conference 2018 Conference Paper

Extreme Low Resolution Activity Recognition With Multi-Siamese Embedding Learning

Michael Ryoo
Kiyoon Kim
Hyun Yang

This paper presents an approach for recognizing human activities from extreme low resolution (e. g. , 16x12) videos. Extreme low resolution recognition is not only necessary for analyzing actions at a distance but also is crucial for enabling privacy-preserving recognition of human activities. We design a new two-stream multi-Siamese convolutional neural network. The idea is to explicitly capture the inherent property of low resolution (LR) videos that two images originated from the exact same scene often have totally different pixel values depending on their LR transformations. Our approach learns the shared embedding space that maps LR videos with the same content to the same location regardless of their transformations. We experimentally conﬁrm that our approach of jointly learning such transform robust LR video representation and the classiﬁer outperforms the previous state-of-theart low resolution recognition approaches on two public standard datasets by a meaningful margin.

PDF Details

AAAI Conference 2017 Conference Paper

Privacy-Preserving Human Activity Recognition from Extreme Low Resolution

Michael Ryoo
Brandon Rothrock
Charles Fleming
Hyun Jong Yang

Privacy protection from surreptitious video recordings is an important societal challenge. We desire a computer vision system (e. g. , a robot) that can recognize human activities and assist our daily life, yet ensure that it is not recording video that may invade our privacy. This paper presents a fundamental approach to address such contradicting objectives: human activity recognition while only using extreme low-resolution (e. g. , 16x12) anonymized videos. We introduce the paradigm of inverse super resolution (ISR), the concept of learning the optimal set of image transformations to generate multiple low-resolution (LR) training videos from a single video. Our ISR learns different types of sub-pixel transformations optimized for the activity classiﬁcation, allowing the classiﬁer to best take advantage of existing high-resolution videos (e. g. , YouTube videos) by creating multiple LR training videos tailored for the problem. We experimentally conﬁrm that the paradigm of inverse super resolution is able to beneﬁt activity recognition from extreme low-resolution videos.

PDF Details

AAAI Conference 2017 Conference Paper

Title Learning Latent Subevents in Activity Videos Using Temporal Attention Filters

A. Piergiovanni
Chenyou Fan
Michael Ryoo

In this paper, we newly introduce the concept of temporal attention ﬁlters, and describe how they can be used for human activity recognition from videos. Many high-level activities are often composed of multiple temporal parts (e. g. , sub-events) with different duration/speed, and our objective is to make the model explicitly learn such temporal structure using multiple attention ﬁlters and beneﬁt from them. Our temporal ﬁlters are designed to be fully differentiable, allowing end-of-end training of the temporal ﬁlters together with the underlying frame-based or segment-based convolutional neural network architectures. This paper presents an approach of learning a set of optimal static temporal attention ﬁlters to be shared across different videos, and extends this approach to dynamically adjust attention ﬁlters per testing video using recurrent long short-term memory networks (LSTMs). This allows our temporal attention ﬁlters to learn latent sub-events speciﬁc to each activity. We experimentally conﬁrm that the proposed concept of temporal attention ﬁlters beneﬁts the activity recognition, and we visualize the learned latent sub-events.

PDF Details