The local-LLM agent¶
English | Japanese
The agent (scripts/orheo_agent) turns a request in plain language into runs
on the multi-fidelity ladder. A local LLM reads the request and fills a few
bounded answers; plain Python decides everything else: which tier, which
queue, when to stop, whether a result is valid. This page shows what the LLM
is asked, how a wrong answer is caught, what was measured on five local
models, and what is still open. The design principles are on
Architecture; the model comparison and the cloud-teacher
scheme are on Local LLM comparison.
What the LLM is asked¶
| call | the LLM answers | bounded by |
|---|---|---|
| INTERPRET | which case and which quantities are asked | the closed case list of the catalogue and a closed vocabulary of quantities |
| SPEC | the settings of the run | the keys, types and ranges of the case schema |
| REPORT | the explanation of the result | "quote the numbers verbatim, do not invent" |
| NEXT | what to do next | a closed menu |
| DIAGNOSE | why a run failed | the playbook of known failures |
Every call asks ollama for JSON that follows a schema, at temperature 0 with a fixed seed. The answer is validated; a failed check is sent back with the error, up to three times, and then a fixed default is used. A weak model can lower the quality of an answer, but it cannot stop the loop. The LLM never chooses a queue or a wall time, never edits code, never reads raw solver output and never decides a verdict. It does not write OpenFOAM dictionaries either: those are filled from vetted skeletons (see Multi-fidelity).
Five local models, measured¶
The evaluation batteries (scripts/llm_eval) were run on a GH200 with ollama
(2026-07-15). Every model answers the easy questions; the models differ only
when they must diagnose a failed run from its log, and on paraphrased
requests:

So the calls are routed per call site (2026-08-21): only DIAGNOSE goes to the large model. A 96 GB GH200 holds a 14b and a 72b model resident.
What the LLM sees¶
The driver does not search documents. It gives the LLM a small context pack
(scripts/orheo_agent/packs/), chosen by the state of the run: the case
catalogue, the case schemas, the scheduler notes, the playbook of failures and
worked examples. build_pack.py checks that the pack and the schemas agree.
A separate retrieval index for people (scripts/rag/, bge-m3 embeddings)
answers questions about the code and the records; on 20 test questions the
right chunk came first in 19 and within the top three in all 20. Teaching the
7b model through retrieval raised its score on the driver battery from 5/10 to
9/10.
Modules and gates¶
Physics is written as modules with one contract (parameters, state, init, step) and one golden ctest each, so that the agent adds a case by composing modules instead of copying a 300-line benchmark. Of the 80 gallery cases, 9 have a direct gate, 28 are covered through their modules and 43 have none (see Validation status).
Tried and set aside¶
- A PINN tier (a network trained on run data to answer between runs) was measured on 2026-07-15 and not adopted: it missed its calibration gate (0.60 at best against ≥ 0.80).
- Fine-tuning a local model was cancelled; the context pack and the retrieval index carry the knowledge instead.
- Geometry from a sketch: only the template lane is built (eight shape classes from text, and STL files passed through); reading a drawing with a vision model is not built.
Still open¶
- REPORT is checked for length only; its numbers are not compared with the run's numbers yet.
- The station's default model is still
qwen3:14b, while every measurement above used qwen2.5. - Diagnosing a failure from a long log: three of six on every model.
build_pack.pycurrently reports eight mismatches between the catalogue and the steady-RANS schemas.- The Nsight profiling station and the demo rehearsal scripts of the design records were not built.
Design records¶
Frozen; they are not updated and are not translated.
- local-llm-ops-design.md — the agent and how the LLM is used
- module-architecture-design.md — modules and golden gates
- opencae-demo-design.md — the OpenCAE demo
- pinn-tier-design.md — the PINN tier
- sketch-to-geometry-design.md — geometry from a sketch
- lbm-landscape-and-methods.md — other LBM codes and their licences