- Maintenance and reliability
- Knowledge retrieval and reporting
- Language models
- Open and synthetic datasets
- Evaluation and benchmarks
Does searching a site’s own work orders help a local LLM as much as a glossary?
MaintNorm · No. 13’s 300 held-out work orders · Qwen 3.8 27B · an offline SAG-Wiki search of each company’s 3,200 other notes · every answer read blind
No. 13 gave a local LLM each company’s fixed 50-line glossary. Here the same model read the same 300 notes with ten of the company’s other work orders instead, picked for each note by SAG-Wiki’s search with what an annotator said each meant, and the glossary ran again the same day.
What we found
- More on this runBy the final verdicts, 290 of 300 notes kept their meaning with the search and 273 with the glossary the same day: 22 only with the search, 5 only with the glossary. No. 13’s glossary run kept 283.
- The glossary varies tooRun on two days, the glossary kept 283 and 273 notes, 20 changing verdict: whether the search usually does better, and by how much, is open.
- Fewer misreadingsOf the 508 shorthand items scored under No. 13’s rules, 7 were read wrong with the search and 12 with the glossary the same day.
- Borrowed wordsThe graders flagged three answers with the search for a word added from one of the other notes, such as “haul truck”, and none with the glossary, whose one invented step, a lockout, they scored as a misreading.
Method
The same notes and model, a search in the glossary’s place
- RealThe same 300 work ordersMaintNorm’s notes from three Australian mining companies (MIT licence; asset numbers are mock values): No. 13’s 300 held-out notes, unchanged.
- SearchedThe company’s other notesSAG-Wiki 0.3.11 deep search over its 3,200 training notes (MaintNorm’s split, not dated), each with the annotator’s reading; Qwen3-Reranker-0.6B judging; loopback only; ten a note.
- ModelQwen 3.8 27B, as in No. 13The same server and prompt, the ten passages in the glossary’s place. The glossary version ran again the same day.
- ScoredBlind, then adjudicatedA note’s meaning is kept when the plain-English reading says what the note says, nothing misread, dropped or added (shorthand left visibly unread does not break it). 600 answers and 40 of No. 13’s, shuffled under random ids, read by four Claude readers under No. 13’s rules (36 of the 40 matched No. 13). Nine verdicts and four single readings changed after unblinding, each to No. 13’s ruling or a written rule, all listed.
Limits: one model, one answer per note and version; one annotator’s key; MaintNorm public since 2024, so the model may have met these notes; the search reads the annotator’s reading of 3,200 other notes, which a site would have to build; AI readers. 47 test notes repeat another note once asset numbers are set aside (55 by a broader check); the result holds without them. Prompt, notes, every answer and verdict, and the scripts: the article’s public repository.
01 · What the model saw
Ten other notes, or fifty glossary lines
The glossary is the same 50 lines for every note. The search picks ten of the company’s other notes for each one, as typed and as the annotator read them. Counted against the key, each put the meaning of about the same share of the shorthand in front of the model.
Supplied: the shorthand’s form and the key’s meaning in one glossary line, or in one other note the search returned. Both supplied 327 pieces; only the search, 46; only the glossary, 42. Prompts were similar in size: a median of 578 tokens with the search, 621 with the glossary.
02 · The result
Fewer misreadings with the search
The 508 shorthand items scored under No. 13’s rules, each against MaintNorm’s key and then checked blind by the AI readers; shorthand the key leaves as written is not counted. The grey rows are No. 13’s two recorded readings of the same notes on 20 November; the dark rows were run on 21 November, scored after the adjudication.
All three companies, whole notes: meaning kept in 290 of 300 with the search, 273 with the glossary the same day; No. 13 recorded 283 with the glossary and 228 with no help.
03 · Note by note
Five notes, both answers
Each note as typed, what it means by MaintNorm’s key, the first of the other notes the search found for it, and the model’s plain-English reading with each version. Three the search fixed, one of them adding a word from another job; two it broke.
-
Company C · as typed
no a/c fans
What it means (MaintNorm’s key): no air conditioner fans
What the search found: the first 4 of 10 other notes
no 24volts no a/c= number 24 volts no air conditioneraircon fans not working= air conditioner fans not workingreplace faulty condenser fans= replace faulty condenser fansCheck guards on exhaust fans= check guards on exhaust fans
With the glossaryNumber air conditioner fans
✗ Applies the line “no = number” to “no fans”: the opposite.With the searchNo air conditioner fans.
✓ Right: other notes read “no a/c” as no air conditioner. -
Company A · as typed
XH531 REPLACE CONVERTER SHAFT UNIS
What it means (MaintNorm’s key): XH531 replace converter shaft universal joints
What the search found: the first 4 of 10 other notes
XH531 REPLACE WIPER ASSY SHAFT & LINKAGE= XH531 replace wiper assembly shaft and linkageReplace trans/converter uni= replace transmission / converter universal jointXH531 Refit Spider bearings to Shaft= XH531 refit spider bearings to shaftXH531 Replace rear drive shaft guard= XH531 replace rear drive shaft guard
With the glossaryAsset XH531: Replace converter shaft units.
✗ UNIS is not among the 50 lines: read as units.With the searchXH531 replace torque converter shaft universal joints
✓ Right on UNIS: another note read “converter uni” as universal joint. Adds “torque”, from another job on the same truck. -
Company B · as typed
AL6487 Rotate Pos 3 9 tyres
What it means (MaintNorm’s key): AL6487 rotate position 3 9 tyres
What the search found: the first 4 of 10 other notes
AL6487 C/O pos 1&2 Tyres= AL6487 change out position 1 and 2 tyresblown pos 9 tyre= blown position 9 tyreFN5736 rotate Pos 3 tyre= FN5736 rotate position 3 tyreAL6487 ROTATE POS 6 & 8= AL6487 rotate position 6 and 8
With the glossaryAsset AL6487: Rotate the 9 tyres at Position 3.
✗ Positions 3 and 9 become nine tyres at position 3.With the searchAL6487 rotate position 3 and 9 tyres
✓ Right: other notes read “pos 1&2” as positions 1 and 2. -
Company B · as typed
LD3192 250hr Service
What it means (MaintNorm’s key): LD3192 250 hour service
What the search found: the first 4 of 10 other notes
250H Mech Svce Haul Truck LD3192= 250 hour mechanical service haul truck LD3192500H Mech Svce Haul Truck LD3192= 500 hour mechanical service haul truck LD3192500H Mech Svce Haul Truck LD3192= 500 hour mechanical service haul truck LD31921000H Mech Svce Haul Truck LD3192= 1000 hour mechanical service haul truck LD3192
With the glossaryAsset LD3192: 250 hour service.
✓ Right.With the search250 hour mechanical service for haul truck LD3192.
✗ Adds “mechanical” and “haul truck” from other notes with the same asset number. -
Company A · as typed
PH639 REPL POS3 & 3 BRAKE PACKS
What it means (MaintNorm’s key): PH639 replace position 3 and 3 brake packs
What the search found: the first 4 of 10 other notes
PH639 RETENSION POS 5 & 3 BRAKE PACS= PH639 retension position 5 and 3 brake padsPH639 REPAIR PARK BRAKE FAULT= PH639 repair park brake faultPH639 REPL 6 X BRAKE RELEASE HOSES= PH639 replace 6 x brake release hosesPH639 repair park brake fault= PH639 repair park brake fault
With the glossaryReplace position 3 and 3 brake packs on asset PH639.
✓ Right.With the searchPH639 replace position 3 and 3 brake pads
✗ Copies “brake pads” from another note’s reading of “BRAKE PACS”.
04 · Every note
22 notes only with the search, 5 only with the glossary
One square per note, by whether its meaning came through with each version on the same day. Exact McNemar test, two-sided: p = 0.002. As the readers first judged, blind: 22 against 9, p = 0.03. With the disputed readings counted as lost (four of them kept only by No. 13’s rulings): 286 against 264, 30 against 8.
No answer in either version invented a diagnosis or said work was done that the note did not report.
Against No. 13’s glossary run of 20 November the search kept 290 to 283: 13 notes only with the search, 6 only with that run, not a clear difference (p = 0.17). The glossary itself kept 273 on 21 November against 283 on 20 November, with 20 notes changing verdict. On this run the search did better than the glossary; whether it usually does, and by how much, is open.
05 · At your site
Before a search reads your work orders
- Index the notes you already haveThe search used the same 3,200 annotated notes the glossary’s 50 lines were counted from, with no lines to choose; in both, the annotation is the work.
- Check the readings you indexAnother note read “BRAKE PACS” as brake pads, and the model copied it into a note about brake packs.
- Expect borrowed wordsOther notes with the same asset number put “haul truck” into a note that did not say it. Tell the model the other notes are for shorthand only.
- Mind what each missesThe glossary and the search each covered about 45 pieces of shorthand the other missed; both together were not tested.
- Test on more than one runThe same glossary kept 283 notes one day and 273 the next.
Sources
- Bikaun, T., Hodkiewicz, M. and Liu, W. (2024). MaintNorm, a corpus of maintenance short text with each word normalised. Proceedings of W-NUT 2024, pages 68–78. aclanthology.org/2024.wnut-1.7; data github.com/nlp-tlp/maintnorm.
- SAG-Wiki, open-source document search, MIT licence; CLI 0.3.11 from npm. github.com/Zleap-AI/SAG-Wiki.
- Wu, Y., Liang, X. and Li, G. (2026). SAG2: embedding-free multi-hop retrieval with BM25 and decision models. arXiv:2610.22588 v2. arxiv.org/html/2610.22588v2.
- Models: Qwen3-Reranker-0.6B; Qwen 3.8 27B as 4-bit weights.