$ cat briefs/daily/2026-11-02.md
2026-11-02
November 2, 2026 (Mon)
1 evaluation story · 1 paper pick · 1 tooling update
Top Stories
1. Co-Design Bench separates partial credit from finished work
Across 273 application-revision tasks, the strongest of ten tested endpoints passes an average of 69.9% of criteria per task but fully satisfies only 26.4% of tasks. The evaluation uses source and browser evidence, then challenges positive verdicts with a separate verifier. First submitted 2026-10-26.
Why it matters: an agent can score well while leaving requested changes unfinished; both criterion completion and whole-task completion belong in an evaluation. The abstract does not identify the leading endpoint. → Co-Design Bench — completing every requested revision (source)
Paper Picks
Soft-MTP: retain the useful self-distillation gradient in multi-token prediction while removing a noisy residual. The author reports faster speculative decoding and better next-token learning across 167M to 1.2B parameters, without numerical speedup figures in the abstract. Read it for the training-to-inference connection; deployment-scale gains remain unestablished. → Soft-MTP — isolating a useful multi-token training signal (source)
Watch
- Simon Willison's 2026-10-31 release note says
datasette-llm 0.1a9inherits LLM's default model, avoiding an explicit default-model setting when starting Datasette Agent. This simplifies setup; it does not demonstrate better agent performance. → Agents (LLM Agents) (source)
New in Wiki
- Co-Design Bench — completing every requested revision and Soft-MTP — isolating a useful multi-token training signal add methods, author lists and evidence limits. (evaluation) (training)
Updates
- Eval Harness Configuration records a complexity signal: every tested endpoint completes fewer criteria on repository edits involving three or four change types than on those involving one or two. (source)