eleticle
Tag · Method

Evaluation and benchmarks

Measuring and comparing models on a fixed test with a stated scoring method. 6 entries in the Core Log.

  1. No. 16Nov 2026

    Does searching a site’s own work orders help a local LLM as much as a glossary?

    The same local LLM read No. 13's 300 held-out mining work orders with ten of the company's other work orders, found for each by an offline SAG-Wiki search, in place of a 50-line glossary. Meaning came through in 290 notes with the search, 273 with the glossary run the same day (22 against 5 kept by only one) and 283 in No. 13's glossary run.

  2. No. 15Nov 2026

    Which local model finds the whole answer in a mine’s reports?

    Five public records on Cadia's molybdenum plant, 576 pages, searched in SAG-Wiki with no connection beyond the workstation. The first five passages held the answer in 48 of 49 single-fact searches, but both the feed route and the plant's first filtered concentrate, from two records, in only 6 of 21. Every search returned passages for a shift figure the records do not hold.

  3. No. 13Nov 2026

    Does a site glossary help a local LLM read maintenance work orders?

    Each of three mining companies had a 50-line glossary built from 3,200 of its own notes. Across 300 held-out work orders containing shorthand, a local LLM kept the meaning of 283 with the company's glossary, against 228 without it, and neither version said work was done that the note did not report.

  4. No. 10Nov 2026

    What does a local LLM know about minerals?

    Two local models, 976 questions rebuilt from a published test, grading checked by hand. Flash Next named a geology term from its definition 6 times in 10, but recalled a mineral's basic properties only 4 times in 10.

  5. No. 08Nov 2026

    Does a written skill help a local LLM read a whole plant flowsheet?

    On each of two plant drawings, Qwen 3.8 Flash Next found more streams in all three attempts with a flowsheet-reading skill than in any attempt without it, and drew fewer wrong lines, in about twice the time. The 27B model gained on one drawing of two in each of two runs, not the same one, and in its rerun took about fifteen times as long with the skill as without.

  6. No. 06Nov 2026

    Can a local LLM read a mineral processing plant flowsheet?

    Two local models, four flowsheets. Both read the simple drawings. On a whole plant neither got every line right; the one that zoomed in piece by piece recovered more of the loops.