Author name cluster

Qingyan Guo

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

2 papers

2 author rows

ICLR Conference 2024 Conference Paper

Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers

Qingyan Guo
Rui Wang 0028
Junliang Guo
Bei Li
Kaitao Song
Xu Tan 0003
Guoqing Liu
Jiang Bian 0002

Large Language Models (LLMs) excel in various tasks, but they rely on carefully crafted prompts that often demand substantial human effort. To automate this process, in this paper, we propose a novel framework for discrete prompt optimization, called EvoPrompt, which borrows the idea of evolutionary algorithms (EAs) as they exhibit good performance and fast convergence. To enable EAs to work on discrete prompts, which are natural language expressions that need to be coherent and human-readable, we connect LLMs with EAs. This approach allows us to simultaneously leverage the powerful language processing capabilities of LLMs and the efficient optimization performance of EAs. Specifically, abstaining from any gradients or parameters, EvoPrompt starts from a population of prompts and iteratively generates new prompts with LLMs based on the evolutionary operators, improving the population based on the development set. We optimize prompts for both closed- and open-source LLMs including GPT-3.5 and Alpaca, on 31 datasets covering language understanding, generation tasks, as well as BIG-Bench Hard (BBH) tasks. EvoPrompt significantly outperforms human-engineered prompts and existing methods for automatic prompt generation (e.g., up to 25% on BBH). Furthermore, EvoPrompt demonstrates that connecting LLMs with EAs creates synergies, which could inspire further research on the combination of LLMs and conventional algorithms.

Details

NeurIPS Conference 2024 Conference Paper

Predictor-Corrector Enhanced Transformers with Exponential Moving Average Coefficient Learning

Bei Li
Tong Zheng
Rui Wang
Jiahao Liu
Qingyan Guo
Junliang Guo
Xu Tan
Tong Xiao

Residual networks, as discrete approximations of Ordinary Differential Equations (ODEs), have inspired significant advancements in neural network design, including multistep methods, high-order methods, and multi-particle dynamical systems. The precision of the solution to ODEs significantly affects parameter optimization, thereby impacting model performance. In this work, we present a series of advanced explorations of Transformer architecture design to minimize the error compared to the true ``solution. '' First, we introduce a predictor-corrector learning framework to minimize truncation errors, which consists of a high-order predictor and a multistep corrector. Second, we propose an exponential moving average-based coefficient learning method to strengthen our higher-order predictor. Extensive experiments on large-scale machine translation, abstractive summarization, language modeling, and natural language understanding benchmarks demonstrate the superiority of our approach. On the WMT'14 English-German and English-French tasks, our model achieved BLEU scores of 30. 95 and 44. 27, respectively. Furthermore, on the OPUS multilingual machine translation task, our model surpasses a robust 3. 8B DeepNet by an average of 2. 9 SacreBLEU, using only 1/3 parameters. Notably, it also beats LLama models by 5. 7 accuracy points on the LM Harness Evaluation.

PDF Details DOI