$ cat wiki/papers/2026/2608.18565-semaplc.md

SemaPLC: A Project-Grounded, Verification-Gated Agent Harness for PLC Code Generation (arXiv:2608.18565)

paperupdated 2026-08-21created 2026-08-21

agents harness coding verification industrial open-source

TL;DR

An agent harness for industrial PLC code whose defining rule is that a task is complete only when logged external checks say so — never when the model judges its own output adequate. The gap it exposes is the finding: static scores separate methods by under 10 points, while live-runtime dynamic behaviour separates them 22.4–31.4 against SemaPLC's 52.2 (source).

Authors & Org

Not obtainable in full. arxiv.org is EGRESS_BLOCKED from this environment and the paper was not read; the HuggingFace Daily Papers snapshot carries title, id, date and abstract only (source).

The abstract states the code is open-sourced at https://github.com/midea-ai/SemaPLC — the organisation segment names Midea, recorded as what the URL says rather than as a confirmed affiliation. Not visited; github.com is under the same egress block.

Listed on HuggingFace Daily Papers, 2026-08-21, 110 upvotes — that community's popularity signal and nothing more (source).

Method

Programmable logic controllers run industrial plants. LLMs can already generate independent program organization units (POUs) for them; whether that logic integrates into an existing project and then runs correctly has been checked only in limited tests.

SemaPLC is described as assembled from conventional tools — the contribution is not a new component but a strict completion rule:

the task is complete only when logged external checks confirm it, not when the model judges its own output adequate.

Three layers of check, in increasing strength:

Layer	What it verifies
Specification	the generated logic matches what was asked
Compilation	it compiles — and, on the project track, compiles inside a real project
Dynamic behaviour	generated and reference logic are deployed to a live PLC runtime and their executed traces compared
Two evaluation tracks: 117 independent-POU tasks matching existing benchmarks,
and a project-context track of 65 tasks whose generated logic must compile and
run inside a real project.

Results

Item	Figure
Independent-POU track	highest strict verified pass rate on all seven models, 72.6% mean
Project-context track	highest mean on integrated compilation, static behaviour and dynamic behaviour
Static behaviour, all methods	within 10 points of one another
Dynamic behaviour, baselines	22.4 → 31.4
Dynamic behaviour, SemaPLC	52.2
The paper's own conclusion, quoted because it is the transferable part:
**"Execution, not static scoring, is the faithful test of whether generated control
logic actually works."**

What the abstract does not give: the seven models' identities, the unit of the dynamic score, per-model breakdowns, and what fraction of the gain comes from the completion rule versus the assembled tooling.

Significance

The measurement finding outlives the domain. Take away PLCs and the result is: a scoring method that all systems pass equally is not measuring the thing. Static scores compressed seven models and several methods into a 10-point band; running the code spread them across 30 points. Any benchmark whose scoring stops at "does it look right" is at risk of the same compression, and most code benchmarks this wiki holds score against tests rather than against executed traces compared to a reference.

It is the strongest instance yet of a pattern this wiki has been assembling all week. Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development (arXiv:2608.13417) argued for within-run metrics over final scores; Apodex Discovery: Reality Benchmarks and Environments for Evaluating and Building Discoverative Artificial Intelligence (arXiv:2608.11341) scored Tools/Repair/Alternatives independently of final-task success. SemaPLC goes further by making the external check the termination condition rather than a richer report — the agent cannot stop until something outside it agrees. That is a harness-design claim, and it sits opposite ASI-Bench: At the Dawn of Artificial Superintelligence (arXiv:2608.17271)'s finding that agents collapse when human methodological guidance is withdrawn: an external verifier is guidance that does not need a human in the loop, for the subset of tasks where one can be built.

The domain is the limit of the result, and it is a real one. PLC logic has a live runtime, a reference implementation and comparable execution traces. Research tasks, the setting where How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks (arXiv:2608.14905) found a model-level deficit across all eight harness–model combinations, have none of the three. Verification-gating works precisely where verification is cheap, which is where agents were already most reliable.

Also worth noting for Agents (LLM Agents): this is an agent result from industrial automation rather than from a frontier lab or an AI-native startup, and its numbers are lower and its checks stricter than the software-agent literature's.

Open Questions

Which seven models? A "highest on all seven" claim is uninterpretable without them, and one of the seven being far weaker than the rest would change the reading.
What is 52.2 out of? The dynamic score's unit and ceiling are unstated, so the gap to 31.4 cannot be converted into a rate.
How much is the rule and how much is the tooling? The paper credits a completion rule while shipping an assembled harness. No ablation is in the abstract.
Does trace comparison generalise? It requires a reference implementation. For novel logic — the case that motivates code generation — there is none.
Cost. Deploying every candidate to a live runtime is expensive; no figure is given.
Author list, affiliation, licence — the GitHub organisation suggests Midea; nothing read confirms it.

Cite

SemaPLC: A Project-Grounded, Verification-Gated Agent Harness for PLC Code Generation (2026). arXiv:2608.18565.

Referenced by

Agents (LLM Agents)Eval Harness Configuration OmniScientist: An Omni-Modal Omni-Discipline AI Scientist (arXiv:2608.13558)