Agents (LLM Agents)

Definition

Systems that place an LLM at their core as the controller to perform multi-step planning + tool use + environment interaction. The ability to carry out sequential tasks beyond a single response.

Representative patterns:

• ReAct (Reason → Act → Observe loop)
• Reflexion (self-critique + retry)
• Tool calling (function calling, MCP)
• Code generation as control flow

Why It Matters

• Squarely at the center of personal interests (1.5x weighting)
• The thickest current in the 2026 AI industry — nearly every frontier lab has an agent line
• The inflection point in the LLM's transition from "text generator" to "general-purpose worker"

State of the Art (2026-05)

Measured on long-horizon R&D: "engineering optimizers, not researchers" (2026-08-18)

Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development (arXiv:2608.13417) evaluates 7 frontier models on 36 long-horizon AI-research-and-development tasks using rule-based within-run metrics rather than final scores, and states the position plainly: current agents "operate more like engineering optimizers than fully autonomous researchers" (source).

The three findings, as reported:

• They can formulate and implement practical solutions.
• Their strongest solutions adapt or combine established techniques; genuine methodological novelty remains rare.
• Performance varies substantially across runs, and experience reuse can help or mislead later decisions.

This is the most direct measurement this page carries of where autonomy actually sits on multi-day work, and it is taken from process evidence rather than a leaderboard. No numbers appear in the abstract and the seven models are not named, so every claim here is the paper's, stated qualitatively. See Eval Harness Configuration for what it implies about the harness results collected there.

Industry side

• Claude Opus 4.7 — Anthropic, emphasizing agents/multi-step
• GPT-Rosalind + Codex — OpenAI, OpenAI's software engineering agent
• Muse Spark (1.0 / 1.1) — Meta MSL, emphasizing "personal"
• Mistral Medium 3.5 + Vibe remote agents — European camp
• Google DeepMind AlphaEvolve — coding agent

Agent Platforms & Infra (2026-06)

• Microsoft Build 2026 Agent Stack (confirmed in the 2026-06-02 keynote): the entire agent execution stack launched simultaneously.
  • Windows Agent Framework 1.0 — MIT-licensed open source; supports Windows 11/365/Azure Arc; built-in human approval queue (actions requiring permissions must be approved by a human); model-agnostic
  • Azure Agent Mesh — a federated multi-agent execution platform spanning on-prem Windows/Windows 365/Azure Arc edge; uses the same API as local; automatic routing based on latency/GPU availability; GA target Q4 2026
  • GitHub Copilot App — an agent-native standalone desktop app (macOS/Windows)
  • VS Code multi-agent GA — on the day of Build
  • Copilot Workspace GA (Enterprise) — an autonomous agent for bug fixes/writing tests/opening PRs
  • Azure AI Foundry — first-class support for Claude, Mistral, Llama 4, DeepSeek → multi-model agent orchestration
  • Project Polaris — MoE coding agent, GitHub Copilot default model GA 2026-08
  • MAI-Code-1 / MAI-Code-1-Flash — 5B class, GA across all Copilot tiers on the day
  • MAI-Thinking-1 — 35B active, specialized for reasoning/orchestration → Microsoft (source)

Non-Developer Agents: Codex for Every Role (2026-06-02)

OpenAI announced role-specific plugin expansion of Codex (source):

• 6 role-specific plugins with 62 popular apps + 110 skills (Analyst, Marketer, Operator, Designer, Researcher, Investor/Banker)
• Codex Sites (preview): creates interactive hosted web apps from text → dashboards, planners, review workspaces
• Non-developers: 20% of Codex users, growing 3× faster than developers
• Coming: Corporate Finance, PE, Marketing Strategy, Strategy Consulting, Legal plugins
• Significance: "AI agent" is crossing the developer ↔ non-developer boundary; software engineering agents generalizing to knowledge work. Most capable agent (Codex) now positioned as universal knowledge worker tool.

→ OpenAI

Multi-Agent Harness Design for Long-Running Tasks (Anthropic, April 2026 —

Anthropic Engineering Blog describes a three-agent architecture enabling multi-hour autonomous coding/frontend design sessions (source):

• Planner — expands a short product prompt into full spec
• Generator — builds the application
• Evaluator — uses Playwright MCP to test behavior against contracts (separating creation from critique)
• Context reset technique — completely clears context window at overflow, passes structured handoff (state + next steps) to fresh agent
• Results: solo run $9/20 min vs. full harness $200/6 hr → significantly more complete product
• Related: Anthropic 2026 Agentic Coding Trends Report (Jan 2026) — "delegation gap": developers delegate only 0-20% of tasks; harness design addresses quality problem of longer autonomous runs

Significance: Context reset + structured handoff is an emerging pattern for scaling agent runs beyond single context windows — enables production-quality outputs without manual re-intervention.

→ Anthropic

Computer Use (GUI Agents)

• Anthropic + Vercept (Feb 2026): the Vercept acquisition strengthens Claude computer use. OSWorld <15% (2024-Q4) → 72.5% (2026-05). Cloud-hosted MacBook remote-control architecture. → Anthropic (source)
• Browser and desktop GUI manipulation is emerging as an important execution environment for agents — general-purpose computer use beyond code execution

Research side

• Agentic Reinforcement Learning — the RL learning direction
• Embodied Agents — extension into the physical world
• MCP (Model Context Protocol) — Anthropic standard → broad adoption (default Microsoft 365 Copilot integration 2026-01)

Sub-concepts

• Agentic Reinforcement Learning — the RL aspect
• Embodied Agents — the physical-world aspect
• tool-use, mcp, reflection separate pages TBD

Persistent Memory: OpenAI Dreaming V3 (2026-06-04)

OpenAI released Dreaming V3 — a background memory synthesis architecture that represents the most significant advance in agent/user memory since the original ChatGPT memory rollout (source):

• Mechanism: Background process continuously synthesizes salient facts, preferences, and temporal context from all conversations — no explicit user instruction required
• Temporal awareness: Auto-updates time-sensitive memories (e.g., past events are revised from future tense to past tense)
• Replaces: The explicit "saved memories" list as the standalone memory foundation
• Performance: Factual recall 41.5% (2024) → 82.8% (2026); preference/time-sensitive accuracy in low-70s
• Compute: 5× reduction in memory-related inference compute → enables Free-tier rollout
• Rollout: June 4, Plus/Pro US first; Free and global to follow

Disambiguation: OpenAI "Dreaming" (personalization memory for ChatGPT users) vs Anthropic "Dreaming" (agents reviewing past sessions to form procedural memory for self-improvement — see Claude Managed Agents). Same name, different mechanism.

Agent relevance: Persistent user memory is a foundational requirement for continuity in long-running agents. Dreaming V3 solves this at the product layer for consumer ChatGPT; the architectural pattern (background synthesis vs. explicit storage) is the key technique.

→ OpenAI

Google ADK 2.0 GA + Agents CLI (2026-06-30)

Google's ADK reached GA with graph workflows (fan-out/fan-in, loops, state management, human-in-the-loop) and a collaborative Task API for agent-to-agent delegation. New Agents CLI covers the full lifecycle in one tool (scaffold → evals → deploy → observability → publishing). Compatible with Claude Code, Cursor, Gemini CLI. Addresses the key eval gap (89% teams have observability, only 52% have evals). → Google ADK (Agent Development Kit), Google DeepMind (source)

Open Problems

• Long-horizon stability — drift / failure accumulation in 50+ step tasks
• Verification — can an agent verify the results of its own work?
• Memory — context limits vs. cumulative learning (Dreaming V3 offers a consumer-layer solution). As of 2026-08-20 this is measured and the answer is regime-dependent, not a ranking: Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents (arXiv:2608.15008) finds no substrate dominates, and more retrieval helps factual QA while hurting sequential decision-making
• Cost — token cost per agent run vs. gain
• Safety — side effects of autonomous action

Key Papers / Events

• Demystifying Agent Skills: Why They Work — Until They Don't (arXiv:2608.14036) (arXiv:2608.14036, 2026-08-14) — what a skill actually does, measured rather than assumed. From 8,135 normalised trial records and 238 open-coded labels: procedural anchoring accounts for 65.7% of skill cases against 4.5% for explicit knowledge injection, so a skill library behaves like a runbook, not a knowledge base. Skills beat Workflow Memory by +6.06 points in matched comparisons. The separate failure is retrieval: actual-use precision falls 29.6% → 3.3% as the pool grows from 5 to 100 — every skill result on this wiki is reported at one, small, pool size (source).

• Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents (arXiv:2608.15008) (arXiv:2608.15008, 2026-08-15) — the Memory open problem below, measured across substrates. One harness, 26 metrics, 3 backbones, 4 suites, every common substrate from dense indices to parametric updates: no substrate consistently dominates, broad retrieval benefits long-context factual QA, and excessive retrieval harms sequential decision-making by shifting attention off action-critical context. Substrates that work at moderate history lengths can turn costly or brittle at longer ones. Its proposal is substrate routing rather than a winner (source).

• AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses (arXiv:2608.12307) (arXiv:2608.12307, 2026-08-12) — the scaffolding can be built by another model, at test time, and it roughly doubles the weaker model's score. A stronger builder constructs an inference-time harness for a weaker target, refining it over multiple rounds against 5% of the data held out as validation; average performance across four Theory-of-Mind benchmarks rises 0.49 → 0.91 with no parameter updates. Salesforce AI Research and UIUC. The paper was not read — arxiv.org is blocked from this environment and the HuggingFace Daily snapshot carries no abstract for this entry, so which models, which four benchmarks and what dispersion are all unknown here (source). It bears directly on the "agent = LLM + scaffolding" vs "agents require separate training" debate below: this is evidence for the first, obtained without touching weights.

• Auto mode became the Claude Code default on 2026-08-14, on the schedule announced 2026-08-07. The figure this run adds is the paired one: head to head, the classifier blocked 800 commands the human testers approved, while humans blocked 6 the classifier allowed. Anthropic also states it will not charge for the classifier's tokens. No false-positive rate has been published, which is the number that would say what the 800 cost (source).

• OpenART: Scaling Agent Red Teaming via Open-Ended Environment Evolution (arXiv:2608.00677) (arXiv:2608.00677, 2026-08) — red-team the environment, not the prompt. An arena of 10,000+ validated stateful scenarios across 50 domains, drawn from 500,000+ tools and skills, with a median of 97 tool calls per task and 75 agent-model configurations. The argument: agent risk accumulates through shared state reused across long-horizon workflows, and short static safety benchmarks cannot see it. No results were readable — the abstract carries scale figures only, and arxiv.org is blocked from this environment (source). Fudan University, Shanghai AI Laboratory and XSafeAI — an academic consortium, where the agent-safety evaluations this wiki holds are almost entirely first-party.

• Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents — 2026-07-29: a foundation GUI agent across mobile, computer, browser and DeepSearch, reporting 82.1% on MobileWorld against Opus 4.8 at +14.6 points, and describing a harness that initiates rather than waits to be invoked (source)

• Scaling the Horizon, Not the Parameters: Reaching Trillion-Parameter Performance with a 35B Agent — 2026-06-30: horizon scaling beats parameter scaling for agentic tasks (35B MoE matches 1T-class models via 45K-token trajectories)

• Self-Distilled Agentic Reinforcement Learning — 2026-05 latest

Regulating the interaction surface: TC260 draft (2026-07)

China's TC260 put out a Cybersecurity Standards Practice Guide — Security Requirements for AI Agent Interaction (Draft for Public Comment v0.23), covering agent-to-agent and agent-to-tool interaction across installation, configuration, use and removal, plus supply-chain controls and employees' unapproved "shadow agents" (source).

It is the first governance document tracked here that regulates what agents do to each other rather than what a model is. The requirements read like IT asset management applied to agents: security assessment before use, hardening before deployment, strict permission control throughout the lifecycle, secure data erasure on decommissioning. A separate mandatory national standard on AI agent safety is at the drafting-plan stage. Full treatment on AI Governance.

The permission prompt gets measured, and fails (2026-08-07)

Anthropic announced that from 2026-08-14 auto mode becomes the default permission mode for new Claude Code sessions on Pro, Max and Team, putting a classifier rather than the user in front of every tool call (source).

The figures behind it, from 1,053 paying beta testers shown a permission prompt for a clearly dangerous command:

Reviewer	Dangerous commands caught
Auto mode classifier	89%
Human, manual approval	13.6%
Human, after ~50 prompts	closer to 5%
The human-approval prompt has been the industry's default safety control for
agentic coding tools, and this is the first published measurement of how well it
performs. It says the control is weak to begin with and decays with exposure —
habituation rather than inattention, which is a property of the interface and not
of the individual user. That makes the prompt a consent-recording mechanism
rather than a review mechanism, and shifts the design question from "did the
user approve" to "what checked it".

The escape hatch is notable in itself: after three consecutive blocks, or twenty in one session, auto mode reverts control to the user — the classifier defers to the human precisely where it is most likely to be wrong, which is the opposite of the trust ordering the headline figures imply. Anthropic's own X post rounds the human figure to 14% against 13.6% in written coverage; the more precise figure is used here. See Anthropic.

Related Concepts

• Agentic Reinforcement Learning
• Embodied Agents
• Reasoning Models
• MCP — Model Context Protocol — the tool-access protocol agent systems connect through
• Eval Harness Configuration — why an agent benchmark score is a claim about the harness too
• AI Governance — the TC260 agent-interaction draft

Notable Statements

• Andrej Karpathy (2026): "rapidly shifting from 80% manual + 20% agents to 80% agent coding + 20% edits"
• Jim Fan: "LLM acts as 'prefrontal cortex' that orchestrates lower-level control APIs"
• Andrej Karpathy: "it's hard to imagine what creating software at the end of 2026 will look like" (via Sam Altman echo)

Open Debates

• Agent = LLM + scaffolding vs Agents require separate training — part of academia (the agentic RL camp) argues the latter. Part of industry argues the former is sufficient.
• No-gradient orchestration (Jim Fan's position) vs end-to-end neural control (Sergey Levine's camp) — a fork in the learning paradigm.