Bidirectional Representations Augmented Autoregressive Biological Sequence Generation

Autoregressive (AR) models, common in sequence generation, are limited in many biological tasks like de novo peptide sequencing and protein modeling by their unidirectional nature, failing to capture crucial global bidirectional token dependencies. Non-Autoregressive (NAR) models offer holistic, bidirectional representations but face challenges with generative coherence and scalability. To transcend this, we propose a hybrid framework enhancing AR generation by dynamically integrating rich contextual information from non-autoregressive mechanisms. Our approach couples a shared input encoder with two decoders: a non-autoregressive one learning latent bidirectional biological features, and an AR decoder synthesizing the biological sequence by leveraging these bidirectional features. A novel cross-decoder attention module enables the AR decoder to iteratively query and integrate these bidirectional features, enriching its predictions. This synergy is cultivated via a tailored training strategy with importance annealing for balanced objectives and cross-decoder gradient blocking for stable, focused learning. Evaluations on a demanding 9-species benchmark of de novo peptide sequencing task show our model substantially surpasses AR and NAR baselines. It uniquely harmonizes AR stability with NAR contextual awareness, delivering robust, superior performance on diverse downstream data. This research advances biological sequence modeling techniques and contributes a novel architectural paradigm for augmenting AR models with enhanced bidirectional understanding for complex sequence generation. Our code is available on GitHub: https: //github. com/BEAM-Labs/denovo

Authors

• Xiang Zhang Northwest University of China
• Jiaqi Wei
• Zijie Qiu
• Sheng Xu Beihang University
• Zhi Jin School of Computer Science, Peking University MoE Key Lab. of High Confidence Software Technologies(PKU), China
• Zhiqiang Gao Shanghai Artificial Intelligence Laboratory
• Nanqing Dong Shanghai Artificial Intelligence Laboratory
• Siqi Sun Research Institute of Intelligent Complex Systems, Fudan University Shanghai Artificial Intelligence Laboratory

Keywords

No keywords are indexed for this paper.