Author name cluster

Hui Deng

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

5 papers

2 author rows

AAAI Conference 2025 Conference Paper

Deep Non-Rigid Structure-from-Motion Revisited: Canonicalization and Sequence Modeling

Hui Deng
Jiawei Shi
Zhen Qin
Yiran Zhong
Yuchao Dai

Non-Rigid Structure-from-Motion (NRSfM) is a classic 3D vision problem, where a 2D sequence is taken as input to estimate the corresponding 3D sequence. Recently, the deep neural networks have greatly advanced the task of NRSfM. However, existing deep NRSfM methods still have limitations in handling the inherent sequence property and motion ambiguity associated with the NRSfM problem. In this paper, we revisit deep NRSfM from two perspectives to address the limitations of current deep NRSfM methods: (1) canonicalization and (2) sequence modeling. We propose an easy-to-implement per-sequence canonicalization method as opposed to the previous per-dataset canonicalization approaches. With this in mind, we propose a sequence modeling method that combines temporal information and subspace constraint. As a result, we have achieved a more optimal NRSfM reconstruction pipeline compared to previous efforts. The effectiveness of our method is verified by testing the sequence-to-sequence deep NRSfM pipeline with corresponding regularization modules on several commonly used datasets.

PDF Details DOI

EAAI Journal 2025 Journal Article

Photovoltaic system modeling and forecasting techniques: A survey

Chen Luo
Naji Al-Messabi
Zhaoqi Kuang
Changjiang Ma
Ibrahim El-Amin
Hui Deng
Yun Li

Modeling provides engineering science a vital technique in the third paradigm of science. This paper reviews a series of modeling techniques for forecasting solar energy yields of photovoltaic (PV) systems, with comparisons among various aspects of solar photovoltaic forecasting, including forecasting techniques, irradiance models, and PV forecasting software and services. Efficient integration of renewable energy sources, in particular PV systems, to the power gird complement fossil fueled generation in reducing emissions. A pivotal step in this integration is to predict the PV outputs, so as to account for their dynamic energy contribution in generation planning and unit commitment programs. Existing research has mostly been on forecasting solar irradiance figures from a very short term (minutes/hours) to a short term (days/weeks) horizon. The paper highlights the use of advanced intelligent modeling techniques. The results serve as a practical guide to renewable energy researchers and engineers for best available approaches in handling photovoltaic forecasting and renewable power generation.

Details DOI

AAAI Conference 2024 Conference Paper

Exploring Transformer Extrapolation

Zhen Qin
Yiran Zhong
Hui Deng

Length extrapolation has attracted considerable attention recently since it allows transformers to be tested on longer sequences than those used in training. Previous research has shown that this property can be attained by using carefully designed Relative Positional Encodings (RPEs). While these methods perform well on a variety of corpora, the conditions for length extrapolation have yet to be investigated. This paper attempts to determine what types of RPEs allow for length extrapolation through a thorough mathematical and empirical analysis. We discover that a transformer is certain to possess this property as long as the series that corresponds to the RPE's exponential converges. Two practices are derived from the conditions and examined in language modeling tasks on a variety of corpora. As a bonus from the conditions, we derive a new Theoretical Receptive Field (TRF) to measure the receptive field of RPEs without taking any training steps. Extensive experiments are conducted on the Wikitext-103, Books, Github, and WikiBook datasets to demonstrate the viability of our discovered conditions. We also compare TRF to Empirical Receptive Field (ERF) across different models, showing consistently matched trends on these datasets. Code is released at: https://github.com/OpenNLPLab/Rpe.

PDF Details DOI

TMLR Journal 2023 Journal Article

Linearized Relative Positional Encoding

Zhen Qin
Weixuan Sun
Kaiyue Lu
Hui Deng
Dongxu Li
Xiaodong Han
Yuchao Dai
Lingpeng Kong

Relative positional encoding is widely used in vanilla and linear transformers to represent positional information. However, existing encoding methods of a vanilla transformer are not always directly applicable to a linear transformer, because the latter requires a decomposition of the query and key representations into separate kernel functions. Nevertheless, principles for designing encoding methods suitable for linear transformers remain understudied. In this work, we put together a variety of existing linear relative positional encoding approaches under a canonical form and further propose a family of linear relative positional encoding algorithms via unitary transformation. Our formulation leads to a principled framework that can be used to develop new relative positional encoding methods that preserve linear space-time complexity. Equipped with different models, the proposed linearized relative positional encoding (LRPE) family derives effective encoding for various applications. Experiments show that compared with existing methods, LRPE achieves state-of-the-art performance in language modeling, text classification, and image classification. Meanwhile, it emphasizes a general paradigm for designing broadly more relative positional encoding methods that are applicable to linear transformers.

PDF Details

ICLR Conference 2022 Conference Paper

cosFormer: Rethinking Softmax In Attention

Zhen Qin 0003
Weixuan Sun
Hui Deng
Dongxu Li 0003
Yunshen Wei
Baohong Lv
Junjie Yan
Lingpeng Kong

Transformer has shown great successes in natural language processing, computer vision, and audio processing. As one of its core components, the softmax attention helps to capture long-range dependencies yet prohibits its scale-up due to the quadratic space and time complexity to the sequence length. Kernel methods are often adopted to reduce the complexity by approximating the softmax operator. Nevertheless, due to the approximation errors, their performances vary in different tasks/corpus and suffer crucial performance drops when compared with the vanilla softmax attention. In this paper, we propose a linear transformer called cosFormer that can achieve comparable or better accuracy to the vanilla transformer in both casual and cross attentions. cosFormer is based on two key properties of softmax attention: i). non-negativeness of the attention matrix; ii). a non-linear re-weighting scheme that can concentrate the distribution of the attention matrix. As its linear substitute, cosFormer fulfills these properties with a linear operator and a cosine-based distance re-weighting mechanism. Extensive experiments on language modeling and text understanding tasks demonstrate the effectiveness of our method. We further examine our method on long sequences and achieve state-of-the-art performance on the Long-Range Arena benchmark. The source code is available at https://github.com/OpenNLPLab/cosFormer.

Details