RuleTrek: reasoning over rules scattered across files

TL;DR

RuleTrek evaluates whether agents can identify and apply domain rules when scenario evidence is distributed across files. It separates rule identification, deduction and end-to-end reasoning in interactive workspaces. (source)

Authors & Org

Chenhao Yuan, Bohan Yu, Yinhao Xu, Chao Yin, Chenxi Zhou, Zhaoxuan Wang, Shuwen Xu, Shaoping Huang, Haolin Ren, Jiaxiang Liu, Yao Xu, Jun Zhao and Kang Liu. Submitted 2026-10-14; affiliations are not established by the captured abstract page. (source)

Method

A rule-grounded construction pipeline extracts structured rule graphs and incrementally composes rule-specific sub-scenarios while preserving consistency and solvability. The resulting multidomain workspaces test Rule Identification (RI), Rule Deduction (RD) and End-to-End Rule Reasoning (E2E). (source)

Results

Across 11 frontier LLMs, the highest RI and RD scores are 62.47% and 60.43%, respectively. E2E scores range from 26.97% to 55.29%. The abstract does not identify which model achieves each maximum, so the maxima are not presented as one model's profile. (source)

Significance

For Eval Harness Configuration, retrieving the applicable rule and applying it correctly are distinct evaluation targets. The benchmark goes beyond short, self-contained rule puzzles by distributing evidence across an interactive workspace. (source)

Open Questions

How sensitive are the results to navigation tools, context budgets and rule-graph complexity? The abstract does not supply those breakdowns. Full-paper methods and the released code and dataset have not been independently reviewed here. (source)

Cite

• RuleTrek: Benchmarking Agentic Rule Reasoning in Complex Scenarios, arXiv:2610.19698; code and dataset.
• Captured abstract page; selected from HuggingFace Daily Papers, 2026-10-16.