This paper was accepted at the Workshop on Reliable and Responsible Foundation Models (RRFMs) Workshop at ICML 2025.
Uncertainty quantification plays a pivotal role when bringing large language models (LLMs) to end-users. Its primary goal is that an LLM should indicate when it is unsure about an answer it gives. While this has been revealed with numerical certainty scores in the past, we propose to use the rich output space of LLMs, the space of all possible strings, to give a string that describes the uncertainty. In particular, we seek a string that describes the distribution of LLM answers to a question. To measure this, we first propose the SelfReflect distance between a string and a distribution of strings. We verify that it works as intended, and then apply it to study how well modern LLMs can summarize their thoughts, either after sampling responses or even without sampling, just by chain-of-thoughts reasoning.
- ** Work done while at Apple