In a few-shot setting, these explanations can be manually written for a few examples. Prompted this way, the model learns to generate a similar explanation, which is particularly useful on more challenging reasoning problems.
Chain-of-thought prompting can be seen in line of several prior research areas. While explanations have been most commonly used to improve interpretability, Rajani et al. (2019)
train a model to automatically generate explanations during training and inference, achieving a new SOTA on a commonsense reasoning dataset.
In a similar vein, Nye et al. (2020)
train a model to write the intermediate computation steps of an arithmetic problem to a “scratchpad”. For summarization, Narayan et al. (2021)
train a model to generate an entity chain (an ordered sequence of entities mentioned in the reference summary). At test time, the model first generates the entity chain before generating the summary.
There are other ways to improve learning with such intermediate outputs. Wang et al. (2022)
exploit the diversity of reasoning paths by sampling multiple chains of thought and then ensembling the final model predictions. As obtaining explanations for a large number of examples is expensive, Zelikman et al. (2022)
generate explanations for a large dataset by bootstrapping a model in the few-shot setting and only retaining explanations that lead to correct answers.
Using explanations, rationales or a description of reasoning steps works empirically but a more principled theory of how models leverage such rationales is still missing. In particular, it would be interesting to investigate to what extent a model’s reasoning conforms to the reasoning steps preferred by humans (although the model can also be trained to perform more human-like reasoning, similar to InstructGPT
Beyond interpretability, generating an intermediate output enables the user to intervene on a model’s predictions. Narayan et al. (2021)
demonstrate this by removing entities from the entity chain that were not seen in the original input, which improves the faithfulness of the generated summary. As a side-effect, such intermediate-output methods provide an interface and the potential to modulate and steer the predictions of otherwise black-box models. We can thus expect work focusing on whether such rationales truly explain model behaviour, similar to the debate around the explainability of attention
Overall, chain-of-thought prompting and related methods offer a glimpse of the untapped potential of current models. They also present an important and relatively compute-efficient research direction that can bring large improvements on top of state-of-the-art models. In this research, domain expertise is particularly important as it enables the development of strategies, reasoning steps, or alternative input methods that are particularly suited to an application. Prompts also do not need to be restricted to input–output pairs or explanations and can be much richer, including things to avoid, rules of thumb, positive or negative examples, etc as in the schema of Mishra et al. (2022)