🏹 Prompting Techniques
Abstract
This chapter presents brief overivew of some of the popular research papers related to prompt techniques.
[1] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (CoT Prompting)
Overview: The paper introduces a novel prompting technique called "chain of thought prompting," which involves providing a series of intermediate reasoning steps as examples in prompts. Evaluation of three LLMs demonstrates significant improvements in performance across arithmetic, commonsense, and symbolic reasoning tasks.
[2] PAL: Program-aided Language Models
Overview: The paper presents Program-Aided Language models (PAL), an innovative method that combines large language models (LLMs) with a symbolic interpreter for natural language problem-solving. PAL makes use of LLMs to read and generate programs as intermediate reasoning steps, while the actual problem-solving task is given to a Python interpreter.
PAL results in improved accuracy across various mathematical, symbolic, and algorithmic reasoning tasks compared to larger models, achieving state-of-the-art performance in few-shot accuracy on math word problems.
Overview: The paper introduces a new prompting technique called `Program of Thoughts' (PoT) to solve complex numerical reasoning tasks using LLMs. PoT separates reasoning from computation by using language models to express the reasoning process as a program, with computation being performed externally.
Evaluation on multiple datasets shows that PoT outperforms the state-of-the-art method, Chain-of-thoughts prompting (CoT), by an average of 12% in both few-shot and zero-shot settings.
Overview: The paper introduces an interesting zero-shot prompting strategy called Plan-and-Solve (PS) prompting to increase large language models' (LLMs) performance in multi-step reasoning tasks. This approach involves dividing tasks into smaller subtasks and providing detailed instructions, addressing calculation errors and improving reasoning step quality.
Experimental results show that PS prompting consistently outperforms Zero-shot-CoT, is comparable to or exceeds Zero-shot-Program-of-Thought Prompting, and performs similarly to 8-shot CoT prompting in math reasoning tasks.
[5] Large Language Models are Zero-Shot Reasoners (Zero Shot CoT Prompting)
Overview: The paper introduces a novel approach called Zero-shot-CoT that enhances the zero-shot reasoning abilities of LLMs by just adding "Let's think step by step”. This approach significantly improves LLMs' performance on various reasoning tasks, including arithmetics, symbolic reasoning, and logical reasoning, without the need for hand-crafted few-shot examples.
[6] Automatic Chain of Thought Prompting in Large Language Models (Auto CoT Prompting)
Overview: The paper introduces an automatic CoT prompting method: Auto-CoT which constructs demonstrations by sampling questions with diversity. The proposed Auto-CoT consistently matches or exceeds the performance of the CoT prompting technique that requires manual designs of demonstrations
[7] Self-Consistency Improves Chain of Thought Reasoning in Language Models (Self Consistency)
Overview: The paper introduces a novel decoding strategy called self-consistency to enhance CoT prompting, which makes use of diverse reasoning paths for complex tasks. By sampling multiple paths and selecting the most consistent answer, the proposed decoding strategy significantly improves performance on various reasoning benchmarks.
[8] Large Language Models Understand and Can Be Enhanced by Emotional Stimuli (Emotion Prompting)
Overview: This paper investigates the ability of LLMs to understand emotional stimuli and their impact on their performance across various tasks. The authors conduct experiments on 45 tasks using different LLMs and find that LLMs can grasp emotional intelligence and improve their performance with "EmotionPrompt," resulting in significant performance enhancements.
[9] Take a Step Back: Evoking Reasoning Via Abstraction in Large Language Models (Step Back Prompting)
Overview: The paper introduces a new prompting technique called “Step-Back Prompting”, which abstracts high-level concepts and principles from specific instances to improve the reasoning abilities of LLMs.
Experimental results with the PaLM-2 model demonstrate significant performance gains across various reasoning-intensive tasks, such as STEM, Knowledge QA, and Multi-Hop Reasoning, with improvements ranging from 7% to 27%.
[10] Tree of Thoughts: Deliberate Problem Solving with Large Language Models (ToT Prompting)
Overview: The paper introduces a new framework called Tree of Thoughts (ToT) for language model inference, addressing the limitations of token-level, left-to-right decision-making processes in tasks requiring exploration and strategic lookahead.
ToT prompting technique enables language models to consider multiple reasoning paths, self-evaluate choices, and make global decisions, significantly improving their problem-solving abilities.
[11] Large Language Models Are Human-Level Prompt Engineers (Automatic Prompt Engineer)
Overview: The paper proposes Automatic Prompt Engineer (APE), a method for automatically generating and selecting natural language instructions to steer LLMs in various tasks.
APE outperforms LLM baselines and achieves performance comparable to human-generated instructions on most tasks, demonstrating its effectiveness in improving LLM task performance through automated instruction engineering.
[12] ReAct: Synergizing Reasoning and Acting in Language Models (ReAct Prompting)
Overview: In this paper, the authors introduce ReAct, an approach that leverages LLMs to simultaneously generate reasoning traces and task-specific actions. By interleaving these processes, ReAct enhances the synergy between reasoning and acting, improving performance across a range of language understanding and decision-making tasks. It effectively addresses issues like hallucination and error propagation in chain-of-thought reasoning.
[13] Divide and Prompt: Chain of Thought Prompting for Text-to-SQL
Overview: The paper introduces a novel approach, Divide-and-Prompt, which leverages Chain-of-thought (CoT) prompting with large language models (LLMs) to enhance the Text-to-SQL task. By breaking down the task into subtasks and applying CoT prompting, this method improves the LLMs' ability to generate Text-to-SQL statements with higher execution accuracy, addressing a gap in previous research on this topic.
Overview: The paper addresses the issue of misunderstandings between humans and Large Language Models (LLMs) by introducing a method called 'Rephrase and Respond' (RaR) that allows LLMs to rephrase and expand questions posed by humans and provide responses in a single prompt.
The authors propose a two-step variant of RaR for improved performance and demonstrate its effectiveness in various tasks. Additionally, the paper highlights that RaR can complement the popular Chain-of-Thought (CoT) methods and improve LLM performance while contributing to a fair evaluation of LLM capabilities.