Oracle Benchmark:

Existing reasoning datasets and benchmarks fall short in placing LLMs in an interactive, unknown environment. This shortcoming leads to evaluating reasoning in an isolated manner, rather than an integrated, holistic process. We introduce black-box interaction, a novel evaluation paradigm for investigating the integrated, human-like reasoning capability of LLMs, which we term advanced reasoning.

Black-box is defined as a hidden function $f:\mathcal{X}\to \mathcal{Y}$, mapping input $\mathcal{X}=\{x | P(x)\}$ that satisfies predicate $P$ to output $\mathcal{Y}$. LLMs are instructed to interact with the black-box, and the interaction is two-stage: (1). In the exploration stage, LLMs can freely feed any valid input $x$ to the black-box, and will receive corresponding feedback $f(x)$. (2). The evaluation stage starts after reaching given maximum exploration turns. LLMs' comprehension of the black-box is evaluated by comparing their output with the black-box's output on a set of unseen test samples. The practical implementation of a black-box simply involves mapping inputs to outputs based on hidden rules. This simplicity allows it to be generalized across various environments. To facilitate and accelerate the generalization of black-boxes to any scale, type, and level of difficulty, we design a fully automated agentic framework for black-box construction. Three LLM-based modules collaborate to accomplish black-box construction from scratch only with natural language description.

Leveraging this framework, we build the Oracle benchmark, which considers 6 types of black-box task: Code Intent Inference (CII), Circuit Rule Inference (CRI), Physics System Inference (PSI), Encryption Rule Inference (ERI), Interactive Puzzle Inference (IPI), Game Strategy Inference (GSI). The 6 tasks take code, boolean circuit, mechanical system, encryption method, interactive puzzle, opponent's game strategy as black-box respectively. Current benchmark consists of 96 black-boxes, 51 of them are easy black-boxes and 45 are hard.

We evaluate 19 leading proprietary and open-weight LLMs. Overall, reasoning models perform better than chat models. o3 delivers the best performance, ranking first in 5 out of 6 tasks under 10 exploration turns and 4 out of 6 tasks under 20 exploration turns. Furthermore, it achieves an average accuracy exceeding 70% on most easy black-boxes and approximately 40% on most hard ones. Further analysis reveals a critical and universal weakness of LLMs: They lack the high-level planning capability required to develop efficient and adaptive exploration strategies. This deficiency in reasoning prevents effective hypothesis refinement, which consequently compromises the ability to understand complex black-box mechanisms under limited exploration.

Below are black-box interaction examples from 6 tasks Code Intent Inference (CII), Circuit Rule Inference (CRI), Physics System Inference (PSI), Encryption Rule Inference (ERI), Interactive Puzzle Inference (IPI), Game Strategy Inference (GSI) in the Oracle benchmark. These examples cover both exploration and evaluation.

Three LLM-based Modules, Coding LLM, Test LLM, Refinement LLM, cooperate to construct a black-box from scratch only with natural language description. Specifically, Coding LLM generates the complete code for conducting black-box interaction, covering the implementation of black-box and interactive interface between LLMs and black-box. Test LLM is used to interact with the black-box to simulate real interaction scenarios. This simulation covers both exploration and evaluation stage. Refinement LLM is used to check the correctness of generated platform code by combining the interaction log and task rule. Test LLM and Refinement LLM work iteratively until the generated platform code is correct.

Two metrics, accuracy and turn@shot, are used to measure the reasoning ability of LLMs in black-box interaction. The accuracy for each black-box is calculated via $acc = \sum^{K}_{k=1} c^k /K$, where $K$ is the number of test samples and $c^k$ measures the correctness of LLMs' answer. Specifically, accuracy in GSI task is measured by the ratio of actual score to the optimal strategy score. Turn@shot consists of two aspects. Turn denotes to the number of interaction turns for exploration, and shot indicates the number of allowed attempts for each test sample during evaluation. For example, $20$@$2$ means the exploration stage lasts for $20$ turns, and a model has $2$ chances to answer each test sample in evaluation. The best model is supposed to achieve the highest accuracy with the lowest turn@shot.

The following figures show the results of the Oracle benchmark. Models are ranked by the sum of their accuracy on easy and hard black-boxes. Generally speaking, models exhibit similar rankings across 6 tasks. o3, o4-mini, gemini-2.5-pro, claude-3.7-sonnet_thinking, and claude-4-sonnet_thinking achieve competitive performance on all six tasks, and o3 ranks first among all models. When it comes to open-source models, deepseek-r1 achieves the top overall performance. Latest models (e.g., gemini-2.5-flash) perform better than old models (e.g., gemini-2.0-flash). Reasoning models (e.g., claude-4-sonnet_thinking) perform better than conventional chat models (e.g., claude-4-sonnet). While best performing LLMs boast over 80% accuracy on some easy black-box tasks, they still struggle with harder ones, where their accuracy is typically less than half that of their performance on easy tasks.

We conduct an in-depth analysis to understand the performance characteristics and limitations of current models on the Oracle Benchmark. Generally speaking, LLMs struggle to develop efficient and adaptive exploration strategies. We categorize exploration strategies into three tiers. Tier 1: Model can not develop a planned exploration strategy, and explore in a random approach. Tier 2: Model can develop a relatively efficient exploration strategy but fail to optimize it adaptively. Tier 3: Model can adaptively optimize their exploration strategy based on instant feedback, developing a nearly optimal approach. Most LLMs operate at Tier 1. Best-performed reasoning LLMs achieve Tier 2 in some situations. Tier 3 is the domain of human according to Charles Peirce's theory, and we have not yet identified any LLM that can achieve Tier 3 of adaptive strategy planning.