Author name cluster

Guohong Fu

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

6 papers

2 author rows

NeurIPS Conference 2024 Conference Paper

Gated Slot Attention for Efficient Linear-Time Sequence Modeling

Yu Zhang
Songlin Yang
Ruijie Zhu
Yue Zhang
Leyang Cui
Yiqiao Wang
Bolun Wang
Freda Shi

Linear attention Transformers and their gated variants, celebrated for enabling parallel training and efficient recurrent inference, still fall short in recall-intensive tasks compared to traditional Transformers and demand significant resources for training from scratch. This paper introduces Gated Slot Attention (GSA), which enhances Attention with Bounded-memory-Control (ABC) by incorporating a gating mechanism inspired by Gated Linear Attention (GLA). Essentially, GSA comprises a two-layer GLA linked via $\operatorname{softmax}$, utilizing context-aware memory reading and adaptive forgetting to improve memory capacity while maintaining compact recurrent state size. This design greatly enhances both training and inference efficiency through GLA's hardware-efficient training algorithm and reduced state size. Additionally, retaining the $\operatorname{softmax}$ operation is particularly beneficial in ``finetuning pretrained Transformers to RNNs'' (T2R) settings, reducing the need for extensive training from scratch. Extensive experiments confirm GSA's superior performance in scenarios requiring in-context recall and in T2R settings.

PDF Details DOI

ECAI Conference 2024 Conference Paper

PromptCD: Coupled and Decoupled Prompt Learning for Vision-Language Models

Junjie Wu 0005
Mingjie Sun
Chen Gong 0004
Nan Yu
Guohong Fu

Large-scale pre-trained vision-language models (VLMs), like CLIP, have presented striking generalizability for adapting to image classification in a few shot setting. Most existing methods explore a set of learnable tokens, such as prompt learning, on data-efficient utilization for task adaptation. However, they focus on either the coupled-modality property by prompt projection or decoupled-modality characteristic by prompt consistency, which ignores effective interaction between prompts. To model the deep yet sufficient cross-modal interaction and enhance the generalization between both seen and unseen tasks, in this paper, we propose a novel coupled and decoupled prompt learning framework, dubbed PromptCD, for vision-language models. Specifically, we introduce a bi-directional coupled-modality mechanism to intensify the interaction between both vision and language branches. Additionally, we propose mixture consistency to further improve the generalization and discrimination of the models on unseen tasks. The integration of such a mechanism and consistency facilitates the proposed framework adaptation for various downstream tasks. We conduct extensive experiments on 11 image classification datasets under a range of evaluation protocols, including base-to-novel and domain generalization, and cross-dataset recognition. Experimental results demonstrate that our proposed PromptCD overall outperforms state-of-the-art methods.

Details

AIJ Journal 2021 Journal Article

Dependency-based syntax-aware word representations

Meishan Zhang
Zhenghua Li
Guohong Fu
Min Zhang

Dependency syntax has been demonstrated highly useful for a number of natural language processing (NLP) tasks. Typical approaches of utilizing dependency syntax include Tree-RNN and Tree-Linearization, both of which exploit explicit 1-best tree outputs from a well-trained parser as inputs. However, these approaches may suffer from error propagation due to the inevitable errors contained in the 1-best tree outputs. In this work, we propose a novel approach to integrate dependency syntax without using the discrete tree outputs. The key idea is to use the intermediate hidden representations of a well-trained encoder-decoder dependency parser, which are referred to as Dependency-based Syntax-Aware Word Representations (Dep-SAWRs). Then, we simply concatenate such Dep-SAWRs with the conventional context-insensitive word embeddings to compose input word representations, without requiring to modify the model architecture of the downstream tasks. We evaluate the proposed method on four kinds of typical NLP tasks, including sentence classification, sentence matching, sequence labeling and machine translation. Experimental results show that the proposed approach is highly promising. On the one hand, it can utilize dependency syntax effectively, bringing consistently better performance on the four tasks compared with baselines without using syntax. On the other hand, the proposed method can outperform the Tree-RNN and Tree-Linearization approaches in most settings, and meanwhile are highly efficient in syntax integration. In addition, the proposed method would be easily extendable to encoding other structural attributes of language.

Details DOI

AAAI Conference 2019 Conference Paper

Syntax-Aware Neural Semantic Role Labeling

Qingrong Xia
Zhenghua Li
Min Zhang
Meishan Zhang
Guohong Fu
Rui Wang
Luo Si

Semantic role labeling (SRL), also known as shallow semantic parsing, is an important yet challenging task in NLP. Motivated by the close correlation between syntactic and semantic structures, traditional discrete-feature-based SRL approaches make heavy use of syntactic features. In contrast, deep-neural-network-based approaches usually encode the input sentence as a word sequence without considering the syntactic structures. In this work, we investigate several previous approaches for encoding syntactic trees, and make a thorough study on whether extra syntax-aware representations are beneficial for neural SRL models. Experiments on the benchmark CoNLL-2005 dataset show that syntax-aware SRL approaches can effectively improve performance over a strong baseline with external word representations from ELMo. With the extra syntax-aware representations, our approaches achieve new state-of-the-art 85. 6 F1 (single model) and 86. 6 F1 (ensemble) on the test data, outperforming the corresponding strong baselines with ELMo by 0. 8 and 1. 0, respectively. Detailed error analysis are conducted to gain more insights on the investigated approaches.

PDF Details

JAIR Journal 2018 Journal Article

Transition-Based Neural Word Segmentation Using Word-Level Features

Meishan Zhang
Yue Zhang
Guohong Fu

Character-based and word-based methods are two different solutions for Chinese word segmentation, the former exploiting sequence labeling models over characters and the latter using word-level features. Neural models have been exploited for character-based Chinese word segmentation, giving high accuracies by making use of external character embeddings, yet requiring less feature engineering. In this paper, we study a neural model for word-based Chinese word segmentation, by replacing the manually-designed discrete features with neural features in a transition-based word segmentation framework. Experimental results demonstrate that word features lead to comparable performance to the best systems in the literature, and a further combination of discrete and neural features obtains top accuracies on several benchmarks.

PDF Details DOI

IJCAI Conference 2017 Conference Paper

Segmenting Chinese Microtext: Joint Informal-Word Detection and Segmentation with Neural Networks

Meishan Zhang
Guohong Fu
Nan Yu

State-of-the-art Chinese word segmentation systems typically exploit supervised modelstrained on a standard manually-annotated corpus, achieving performances over 95% on a similar standard testing corpus. However, the performances may drop significantly when the same models are applied onto Chinese microtext. One major challenge is the issue of informal words in the microtext. Previous studies show that informal word detection can be helpful for microtext processing. In this work, we investigate it under the neural setting, by proposing a joint segmentation model that integrates the detection of informal words simultaneously. In addition, we generate training corpus for the joint model by using existing corpus automatically. Experimental results show that the proposed model is highly effective for segmentation of Chinese microtext.

PDF Details