Giant Language Fashions (LLMs) are more and more deployed to autonomously remedy real-world duties. A key ingredient for that is the LLM Operate-Calling paradigm, a broadly used method for equipping LLMs with tool-use capabilities. Nevertheless, an LLM calling capabilities incorrectly can have extreme implications, particularly when their results are irreversible, e.g., transferring cash or deleting knowledge. Therefore, it’s of paramount significance to contemplate the LLM’s confidence {that a} operate name solves the duty appropriately previous to executing it. Uncertainty Quantification (UQ) strategies can be utilized to quantify this confidence and stop doubtlessly incorrect operate calls. On this work, we current what’s, to our data, the primary analysis of UQ strategies for LLM Operate-Calling (FC). Whereas multi-sample UQ strategies, reminiscent of Semantic Entropy, present sturdy efficiency for pure language Q&A duties, we discover that within the FC setting, it presents no clear benefit over easy single-sample UQ strategies. Moreover, we discover that the particularities of FC outputs could be leveraged to enhance the efficiency of current UQ strategies on this setting. Particularly, multi-sample UQ strategies profit from clustering FC outputs primarily based on their summary syntax tree parsing, whereas single-sample UQ strategies could be improved by choosing solely semantically significant tokens when calculating logit-based uncertainty scores.
- † College of Oxford
- * Equal contribution
- ‡ Joint senior authorship
