Embodied Agents

Definition

Agents that act through perception and actuation in the physical world or in simulated environments. Unlike pure text-output LLMs, they require a multi-modal perception API + actuation API + skill library structure.

Why It Matters

• Matches agents (1.5x) × intersects with robotics — adjacent to personal interests
• The next frontier beyond applying LLMs to software
• Key 2026 currents: NVIDIA Project GR00T, DeepMind robotics research

State of the Art (2026-07)

• Google DeepMind Gemini Robotics 2 (2026-07-28 / 2026-07-30) — three models: a VLA controlling a full humanoid (legs, torso, arms, multi-finger hands) under one learned policy, an embodied-reasoning VLM, and an on-device VLA. Whole-body manipulation on the Apollo 2 humanoid: 76.3% picking from a shelf, 45.7% from the floor. Fine-motor dexterity on the 22-DoF SharpaWave hand: 92% unscrewing a light bulb, 44% tying a trash bag, 40% sealing a ziplock, 36% screwing a bulb in, 32% dustpan. On-Device 2 adapts to a new embodiment in hours from fewer than 200 demonstrations. Released with ASIMOV-Agentic, an open benchmark for whether the reasoning layer refuses dangerous commands from the acting layer, judges physical feasibility, and asks for human help when uncertain. Significance: the dexterity spread is the state of the field in one table — removing a bulb is near-solved, inserting one is not, and deformable objects sit at the bottom. Note also which layer shipped: the reasoning model is public via the Gemini API, both acting models are early-access only. → Gemini Robotics 2, Gemini Robotics ER 2 (source)

• GRANT / ORS3D — scheduling as the embodied objective (AAAI 2026 Oral; HF Daily #1 on 2026-07-30) — arXiv 2511.19430 defines ORS3D, where the agent minimizes total completion time by running parallelizable subtasks concurrently (clean the sink while the microwave runs), and trains an embodied multimodal LLM with a scheduling token on ORS3D-60K (60K composite tasks, 4K scenes). +30.53% Time Efficiency, +1.38% 3D grounding. Significance: the asymmetry is the point — grounding barely moves, scheduling moves a lot. As per-action reliability improves, the remaining headroom in a multi-step physical task is ordering, not perception. → Cook and Clean Together: Teaching Embodied Agents for Parallel Task Execution (GRANT) (source)

• Mistral Robostral Navigate (2026-07-08) — 8B robotics navigation model; single RGB camera + language prompts only; no depth sensors or LiDAR. Achieves 76.6% R2R-CE validation unseen — new SOTA (+9.7 pts over prior best single-camera; +4.5 pts over best depth/multi-camera). Sim-only training with CISPO online RL (+3.2%). Pointing-based prediction (image coordinates → target + orientation): naturally robust to camera intrinsic changes. Hardware-agnostic. First Mistral physical AI product. Significance: demonstrates that sim-to-real transfer in navigation is now strong enough to beat real-robot-trained depth-sensor systems using only RGB. Lowers hardware bar for deployment. → Robostral Navigate (source)

• NVIDIA ENPIRE — Agentic Robot Self-Improvement on Real Hardware (2026-06-17) published ENPIRE: a fleet of 8 real robots autonomously runs its own research loop (read papers → hypothesize → run trials → verify → rewrite code) — zero humans in the loop. 99% pass@8 on contact-rich tasks. New finding: physical scaling law — 8 parallel robots improve policies superlinearly faster than fewer robots. The first demonstration that the LLM-era data-parallel scaling paradigm transfers to real-world physical manipulation research, not just skill acquisition. Closes the loop on EgoScale: EgoScale → acquire skills from egocentric video; ENPIRE → autonomously improve them. → ENPIRE: Agentic Robot Policy Self-Improvement in the Real World, Jim Fan (source) (arXiv)

• NVIDIA EgoScale — Dexterous Humanoid from Egocentric Video (2026-06-07) ⚡ NEW — Jim Fan's team trained a humanoid with 22-DoF dexterous hands entirely from 20,000+ hours of egocentric human video (no robot teleoperation). Tasks: model car assembly, syringe operation, card sorting, shirt folding. Key result: log-linear scaling law (R² = 0.998) — human video volume → action prediction loss → real-robot success. First strong empirical evidence that the LLM data-scaling paradigm transfers directly to dexterous physical manipulation. Open-sourced: weights, code, dataset, eval set, whitepaper. → Jim Fan (source)

• NVIDIA Cosmos 3 (2026-06-01) ⚡ NEW — The world's first fully open omnimodal physical AI foundation model. 64B (Super), Nano, Edge. MoT architecture: unifies world generation + physical reasoning + action prediction. #1 open model on R-Bench. MIT license. Cosmos Coalition (Agile Robots, Runway, Black Forest Labs, etc.). → Cosmos 3 Super (source)

• Gemini Robotics-ER 1.6 (2026-04-15) — Google DeepMind. Gauge-reading accuracy 23%→93% (agentic vision pipeline). Boston Dynamics Spot collaboration. Crosses the threshold for autonomous inspection on industrial sites. Available via the Gemini API. → Gemini Robotics ER 1.6 (source)

• CaP-X (2026-04) — Open-sourced by NVIDIA's Jim Fan team. "Vibe agents alive in the physical world." Robot arms + humanoids, auto-synthesize skill libraries. (source)

• Project GR00T — NVIDIA humanoid robot foundation model

• DrEureka — LLM writes robot skill training code (Jim Fan team)

• Voyager — Lifelong learning agent in the Minecraft environment

Architecture Patterns

• No-gradient orchestration: The LLM acts as the "prefrontal cortex" for high-level reasoning, while lower-level control is delegated to code-gen
• Skill library: Learned/generated skills are accumulated into a retrievable library
• Perception + actuation API separation — abstracting the environment

Open Problems

• Sim-to-real transfer
• Sample efficiency in real-world RL
• The generalization vs specialization trade-off in skill libraries
• Safety in physical actuation
• Multi-agent coordination

Key Sources

• Cosmos 3 Super — NVIDIA open omnimodal physical AI (2026-06-01, R-Bench #1)
• Jim Fan — Research lead for CaP-X and GR00T
• Gemini Robotics ER 1.6 — DeepMind physical AI for industrial sites, 93% gauge reading
• Project GR00T

Related Concepts

• Agentic Reinforcement Learning — RL-based learning aspect
• Reasoning Models — high-level planning aspect
• NVIDIA — Cosmos 3 + GR00T + CaP-X
• Google DeepMind — Gemini Robotics series
• foundation-agent

Open Debates

• LLM-orchestrated structure vs end-to-end neural control — Jim Fan/NVIDIA strongly favor the former. Part of academia argues for the latter (e.g., the Sergey Levine group).