Skip to content

The local-LLM agent

English | Japanese

« Home

The agent (scripts/orheo_agent) turns a request in plain language into runs on the multi-fidelity ladder. A local LLM reads the request and fills a few bounded answers; plain Python decides everything else: which tier, which queue, when to stop, whether a result is valid. This page shows what the LLM is asked, how a wrong answer is caught, what was measured on five local models, and what is still open. The design principles are on Architecture; the model comparison and the cloud-teacher scheme are on Local LLM comparison.

What the LLM is asked

call the LLM answers bounded by
INTERPRET which case and which quantities are asked the closed case list of the catalogue and a closed vocabulary of quantities
SPEC the settings of the run the keys, types and ranges of the case schema
REPORT the explanation of the result "quote the numbers verbatim, do not invent"
NEXT what to do next a closed menu
DIAGNOSE why a run failed the playbook of known failures

Every call asks ollama for JSON that follows a schema, at temperature 0 with a fixed seed. The answer is validated; a failed check is sent back with the error, up to three times, and then a fixed default is used. A weak model can lower the quality of an answer, but it cannot stop the loop. The LLM never chooses a queue or a wall time, never edits code, never reads raw solver output and never decides a verdict. It does not write OpenFOAM dictionaries either: those are filled from vetted skeletons (see Multi-fidelity).

Five local models, measured

The evaluation batteries (scripts/llm_eval) were run on a GH200 with ollama (2026-07-15). Every model answers the easy questions; the models differ only when they must diagnose a failed run from its log, and on paraphrased requests:

Five local models: accuracy by kind of question, and speed

So the calls are routed per call site (2026-08-21): only DIAGNOSE goes to the large model. A 96 GB GH200 holds a 14b and a 72b model resident.

What the LLM sees

The driver does not search documents. It gives the LLM a small context pack (scripts/orheo_agent/packs/), chosen by the state of the run: the case catalogue, the case schemas, the scheduler notes, the playbook of failures and worked examples. build_pack.py checks that the pack and the schemas agree.

A separate retrieval index for people (scripts/rag/, bge-m3 embeddings) answers questions about the code and the records; on 20 test questions the right chunk came first in 19 and within the top three in all 20. Teaching the 7b model through retrieval raised its score on the driver battery from 5/10 to 9/10.

Modules and gates

Physics is written as modules with one contract (parameters, state, init, step) and one golden ctest each, so that the agent adds a case by composing modules instead of copying a 300-line benchmark. Of the 80 gallery cases, 9 have a direct gate, 28 are covered through their modules and 43 have none (see Validation status).

Tried and set aside

  • A PINN tier (a network trained on run data to answer between runs) was measured on 2026-07-15 and not adopted: it missed its calibration gate (0.60 at best against ≥ 0.80).
  • Fine-tuning a local model was cancelled; the context pack and the retrieval index carry the knowledge instead.
  • Geometry from a sketch: only the template lane is built (eight shape classes from text, and STL files passed through); reading a drawing with a vision model is not built.

Still open

  • REPORT is checked for length only; its numbers are not compared with the run's numbers yet.
  • The station's default model is still qwen3:14b, while every measurement above used qwen2.5.
  • Diagnosing a failure from a long log: three of six on every model.
  • build_pack.py currently reports eight mismatches between the catalogue and the steady-RANS schemas.
  • The Nsight profiling station and the demo rehearsal scripts of the design records were not built.

Design records

Frozen; they are not updated and are not translated.