Author name cluster

Harry Shum

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

10 papers

1 author row

ICLR Conference 2025 Conference Paper

Towards Multiple Character Image Animation Through Enhancing Implicit Decoupling

Jingyun Xue
Hongfa Wang
Qi Tian 0003
Yue Ma 0016
Andong Wang
Zhiyuan Zhao 0002
Shaobo Min
Wenzhe Zhao

Controllable character image animation has a wide range of applications. Although existing studies have consistently improved performance, challenges persist in the field of character image animation, particularly concerning stability in complex backgrounds and tasks involving multiple characters. To address these challenges, we propose a novel multi-condition guided framework for character image animation, employing several well-designed input modules to enhance the implicit decoupling capability of the model. First, the optical flow guider calculates the background optical flow map as guidance information, which enables the model to implicitly learn to decouple the background motion into background constants and background momentum during training, and generate a stable background by setting zero background momentum during inference. Second, the depth order guider calculates the order map of the characters, which transforms the depth information into the positional information of multiple characters. This facilitates the implicit learning of decoupling different characters, especially in accurately separating the occluded body parts of multiple characters. Third, the reference pose map is input to enhance the ability to decouple character texture and pose information in the reference image. Furthermore, to fill the gap of fair evaluation of multi-character image animation, we propose a new benchmark comprising about 4,000 frames. Extensive qualitative and quantitative evaluations demonstrate that our method excels in generating high-quality character animations, especially in scenarios of complex backgrounds and multiple characters.

Details

ICML Conference 2024 Conference Paper

HumanTOMATO: Text-aligned Whole-body Motion Generation

Shunlin Lu
Linghao Chen
Ailing Zeng
Jing Lin
Ruimao Zhang
Lei Zhang 0001
Harry Shum

This work targets a novel text-driven whole-body motion generation task, which takes a given textual description as input and aims at generating high-quality, diverse, and coherent facial expressions, hand gestures, and body motions simultaneously. Previous works on text-driven motion generation tasks mainly have two limitations: they ignore the key role of fine-grained hand and face controlling in vivid whole-body motion generation, and lack a good alignment between text and motion. To address such limitations, we propose a Text-aligned whOle-body Motion generATiOn framework, named HumanTOMATO, which is the first attempt to our knowledge towards applicable holistic motion generation in this research area. To tackle this challenging task, our solution includes two key designs: (1) a Holistic Hierarchical VQ-VAE (aka H${}^{2}$VQ) and a Hierarchical-GPT for fine-grained body and hand motion reconstruction and generation with two structured codebooks; and (2) a pre-trained text-motion-alignment model to help generated motion align with the input textual description explicitly. Comprehensive experiments verify that our model has significant advantages in both the quality of generated motions and their alignment with text.

Details

ICLR Conference 2024 Conference Paper

Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph

Jiashuo Sun
Chengjin Xu
Lumingyuan Tang
Saizhuo Wang
Chen Lin 0001
Yeyun Gong
Lionel M. Ni
Harry Shum

Although large language models (LLMs) have achieved significant success in various tasks, they often struggle with hallucination problems, especially in scenarios requiring deep and responsible reasoning. These issues could be partially addressed by introducing external knowledge graphs (KG) in LLM reasoning. In this paper, we propose a new LLM-KG integrating paradigm ``$\hbox{LLM}\otimes\hbox{KG}$'' which treats the LLM as an agent to interactively explore related entities and relations on KGs and perform reasoning based on the retrieved knowledge. We further implement this paradigm by introducing a new approach called Think-on-Graph (ToG), in which the LLM agent iteratively executes beam search on KG, discovers the most promising reasoning paths, and returns the most likely reasoning results. We use a number of well-designed experiments to examine and illustrate the following advantages of ToG: 1) compared with LLMs, ToG has better deep reasoning power; 2) ToG has the ability of knowledge traceability and knowledge correctability by leveraging LLMs reasoning and expert feedback; 3) ToG provides a flexible plug-and-play framework for different LLMs, KGs and prompting strategies without any additional training cost; 4) the performance of ToG with small LLM models could exceed large LLM such as GPT-4 in certain scenarios and this reduces the cost of LLM deployment and application. As a training-free method with lower computational cost and better generality, ToG achieves overall SOTA in 6 out of 9 datasets where most previous SOTAs rely on additional training.

Details

ICLR Conference 2024 Conference Paper

TOSS: High-quality Text-guided Novel View Synthesis from a Single Image

Yukai Shi
Jianan Wang
He Cao
Boshi Tang
Xianbiao Qi
Tianyu Yang 0003
Yukun Huang
Shilong Liu 0004

In this paper, we present TOSS, which introduces text to the task of novel view synthesis (NVS) from just a single RGB image. While Zero123 has demonstrated impressive zero-shot open-set NVS capabilities, it treats NVS as a pure image-to-image translation problem. This approach suffers from the challengingly under-constrained nature of single-view NVS: the process lacks means of explicit user control and often result in implausible NVS generations. To address this limitation, TOSS uses text as high-level semantic information to constrain the NVS solution space. TOSS fine-tunes text-to-image Stable Diffusion pre-trained on large-scale text-image pairs and introduces modules specifically tailored to image and camera pose conditioning, as well as dedicated training for pose correctness and preservation of fine details. Comprehensive experiments are conducted with results showing that our proposed TOSS outperforms Zero123 with higher-quality NVS results and faster convergence. We further support these results with comprehensive ablations that underscore the effectiveness and potential of the introduced semantic guidance and architecture design.

Details

ICLR Conference 2023 Conference Paper

DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection

Hao Zhang 0097
Feng Li 0040
Shilong Liu 0004
Lei Zhang 0001
Hang Su 0006
Jun Zhu 0001
Lionel M. Ni
Harry Shum

We present DINO (DETR with Improved deNoising anchOr boxes), a strong end-to-end object detector. DINO improves over previous DETR-like models in performance and efficiency by using a contrastive way for denoising training, a look forward twice scheme for box prediction, and a mixed query selection method for anchor initialization. DINO achieves 49.4AP in 12 epochs and 51.3AP in 24 epochs on COCO with a ResNet-50 backbone and multi-scale features, yielding a significant improvement of +6.0AP and +2.7AP, respectively, compared to DN-DETR, the previous best DETR-like model. DINO scales well in both model size and data size. Without bells and whistles, after pre-training on the Objects365 dataset with a SwinL backbone, DINO obtains the best results on both COCO val2017 (63.2AP) and test-dev (63.3AP) with model size under 1 billion parameters. Compared to other models on the leaderboard, DINO significantly reduces its model size and pre-training data size while achieving better results. The code will be available.

Details

IROS Conference 2006 Conference Paper

Human Intention Modeling and Interactive Computer Vision

Harry Shum

For many years, computer vision and robotics researchers have worked hard chasing the illusive goals such as "can the robot find a boy in the scene" or "can your vision system automatically segment the cat from the background". These tasks require a lot of prior knowledge and contextual information, and perhaps more importantly, understanding of human intention. How to model human intention into vision and robotic systems is, however, very challenging and can only be solved through human-computer interaction. In this talk, we propose that many difficult vision tasks can be solved with interactive vision systems, by combining powerful and real-time vision techniques with intuitive and clever user interfaces. We will show two interactive vision systems we developed recently, Lazy Snapping (Siggraph 2004) and Image Completion (Siggraph 2005). Lazy Snapping cuts out an object from a picture using graph cut, while Image Completion recovers unknown region in a picture with belief propagation. A key element in designing such interactive systems is how we model the user's intention using conditional probability (context) and likelihood associated with user interactions. Given how ill-posed most image understanding problems are, it is proposed that interactive computer vision is the paradigm we should focus today's vision research on where the key is the understanding and modeling of human intention.

Details

IROS Conference 1994 Conference Paper

Virtual reality modeling from a sequence of range images

Harry Shum
Katsushi Ikeuchi
Raj Reddy

Virtual reality object modeling from a sequence of range images has been formulated as a problem of principal component analysis with missing data (PCAMD), which can be generalized as a weighted least square (WLS) minimization problem. An efficient algorithm has been devised to solve the problem of PCAMD. After all visible P regions appeared over the whole sequence of F views are segmented and tracked, a 3F/spl times/P normal measurement matrix of surface normals and an F/spl times/P distance measurement matrix of normal distances to the origin are constructed respectively. These two measurement matrices, with possibly many missing elements due to occlusion and mismatching, enable us to formulate multiple view merging as a combination of two WLS problems. By combining information at both the signal level and the algebraic level, a modified Jarvis' march algorithm is proposed to recover the spatial connectivity among all the reconstructed surface patches. Experiments using synthetic data and real range images show that our approach is robust against noise and mismatch. A toy house model from a sequence of real range images is presented. >

Details

IROS Conference 1993 Conference Paper

Implementing model-based variable-structure controllers for robot manipulators with actuator modelling

S. K. Tso
P. L. Law
Yangsheng Xu
Harry Shum

A model-based control scheme for robot manipulators employing a variable structure control law has been found to perform well, provided that the design parameters are carefully chosen. A refinement of the system model of this original scheme in which the actuator dynamics is taken into consideration is studied. Practical experiments are carried out on a commercial revolute-joint robot manipulator.

Details

ICRA Conference 1992 Conference Paper

Adaptive control of space robot system with an attitude controlled base

Yangsheng Xu
Harry Shum
Ju-Jang Lee
Takeo Kanade

The authors discuss adaptive control of a space robot system with an attitude-controlled base on which the robot is attached. An adaptive control scheme in joint space is proposed. Since most tasks are specified in inertia space, instead of joint space, the authors discuss the issues associated to adaptive control in inertia space and identify two potential problems, unavailability of the joint trajectory (since mapping from inertia space trajectory is dynamics-dependent and subject to uncertainty), and nonlinear parameterization in inertia space. For a planar system, the linear parameterization problem is investigated, the design procedure of the controller is illustrated, and the validity and effectiveness of the proposed control scheme are demonstrated. >

Details

ICRA Conference 1991 Conference Paper

Variable structure model reference adaptive control of robot manipulators

S. K. Tso
Yangsheng Xu
Harry Shum

An adaptive control scheme combining the variable structure and model reference methods is presented. With the variable structure technique, all known parameters of the robot system are fully used while the unknown parameters are adaptively adjusted. The overall control system maintains the basic structure of the computer torque controller, but incorporates adaptive components in the system. The method removes the requirement for persistent excitation, essential to traditional adaptive schemes for satisfactory operation. The control algorithm ensures the robustness of the controlled system with respect to disturbance, since the tracking error always converges to zero theoretically, rather than to an ill-defined residual set as in other adaptive schemes. Using this method, the transient response can be prescribed in advance. Thus, all the outstanding issues in adaptive control are directly treated. Simulation analysis for a two-degree-of-freedom robot is conducted to compare the method with the classical model reference method and computed torque method. >

Details