Skip to content

🎾 Step Back Prompting

Abstract

This section covers "Step Back Prompting".

✒️ Overview

Step back prompting technique instructs the LLM to first extract high-level concepts and principles from given details, then use those concepts to guide its reasoning towards the correct solution. This prompting technique is based on the fact that “in critical decision-making, humans find abstraction to be helpful since it provides a broader view of the environment”.

✒️ How it works

Step back prompting involves two steps.

  1. Prompting for Abstraction -

    The first step is to teach LLMs to step back, and derive high-level abstractions such as concepts and first principles from the specific example. This step uses prompts like:

  2. "What are the key principles or concepts at play here?"
  3. "What are the relevant rules or patterns in this situation?"

  4. "How does this situation relate to similar situations I've encountered?"

  5. Utilizing Abstracted Knowledge -

    The second step is to arrive at a solution based on high-level concepts and first principles. Here, the LLM uses the abstracted knowledge gained from the "step back" question to guide its reasoning process. This allows the LLM to ignore irrelevant details and focus on the core aspects of the task.

As shown in the above figure, for a Physics question of “What happens to the pressure, P, of an ideal gas if the temperature is increased by a factor of 2 and the volume is increased by a factor of 8 ?”, the LLM can easily deviate from the first principle of Ideal Gas Law if it directly generates the answer. Instead of directly answering, taking a step back and asking a step-back question helps the model to accurately generate the solution.

✒️ Pros

  • Improved reasoning - Stronger performance on tasks demanding analysis, deduction, and knowledge application.
  • Reduced error rate - Deeper understanding helps to ignore irrelevant details and focus on the core aspects which reduces error rate.

✒️ Cons

  • Prompt design challenge - Crafting effective prompts requires precision to avoid vagueness or bias.
  • Computational cost - Additional abstraction step adds some processing overhead, a potential concern for limited resources.
  • Limited scope - May not be relevant for all LLM applications, especially for text generation.

To summarize, step-back prompting with a two-step approach is a promising technique to enhance the reasoning capabilities of LLMs.

📝 Note: The above image is from "Step Back Prompting" Paper.