eleticle

Core No. 16Logged 21 Nov 2026Code & data →

Does searching a site’s own work orders help a local LLM as much as a glossary?

MaintNorm · No. 13’s 300 held-out work orders · Qwen 3.8 27B · an offline SAG-Wiki search of each company’s 3,200 other notes · every answer read blind

No. 13 gave a local LLM each company’s fixed 50-line glossary. Here the same model read the same 300 notes with ten of the company’s other work orders instead, picked for each note by SAG-Wiki’s search with what an annotator said each meant, and the glossary ran again the same day.

What we found

  • More on this runBy the final verdicts, 290 of 300 notes kept their meaning with the search and 273 with the glossary the same day: 22 only with the search, 5 only with the glossary. No. 13’s glossary run kept 283.
  • The glossary varies tooRun on two days, the glossary kept 283 and 273 notes, 20 changing verdict: whether the search usually does better, and by how much, is open.
  • Fewer misreadingsOf the 508 shorthand items scored under No. 13’s rules, 7 were read wrong with the search and 12 with the glossary the same day.
  • Borrowed wordsThe graders flagged three answers with the search for a word added from one of the other notes, such as “haul truck”, and none with the glossary, whose one invented step, a lockout, they scored as a misreading.
  1. Method
  2. 01What the model saw
  3. 02The result
  4. 03Note by note
  5. 04Every note
  6. 05At your site
  7. Sources

Method

The same notes and model, a search in the glossary’s place

  • RealThe same 300 work ordersMaintNorm’s notes from three Australian mining companies (MIT licence; asset numbers are mock values): No. 13’s 300 held-out notes, unchanged.
  • SearchedThe company’s other notesSAG-Wiki 0.3.11 deep search over its 3,200 training notes (MaintNorm’s split, not dated), each with the annotator’s reading; Qwen3-Reranker-0.6B judging; loopback only; ten a note.
  • ModelQwen 3.8 27B, as in No. 13The same server and prompt, the ten passages in the glossary’s place. The glossary version ran again the same day.
  • ScoredBlind, then adjudicatedA note’s meaning is kept when the plain-English reading says what the note says, nothing misread, dropped or added (shorthand left visibly unread does not break it). 600 answers and 40 of No. 13’s, shuffled under random ids, read by four Claude readers under No. 13’s rules (36 of the 40 matched No. 13). Nine verdicts and four single readings changed after unblinding, each to No. 13’s ruling or a written rule, all listed.

Limits: one model, one answer per note and version; one annotator’s key; MaintNorm public since 2024, so the model may have met these notes; the search reads the annotator’s reading of 3,200 other notes, which a site would have to build; AI readers. 47 test notes repeat another note once asset numbers are set aside (55 by a broader check); the result holds without them. Prompt, notes, every answer and verdict, and the scripts: the article’s public repository.

01 · What the model saw

Ten other notes, or fifty glossary lines

The glossary is the same 50 lines for every note. The search picks ten of the company’s other notes for each one, as typed and as the annotator read them. Counted against the key, each put the meaning of about the same share of the shorthand in front of the model.

508 pieces of shorthand in the 300 notessearch 373 · glossary 369
All three, 508 pieces of shorthand: the glossary gives the key's meaning for 369, the search passages for 373, both for 327; Company A, 157 pieces of shorthand: the glossary gives the key's meaning for 118, the search passages for 111, both for 103; Company B, 203 pieces of shorthand: the glossary gives the key's meaning for 162, the search passages for 164, both for 154; Company C, 148 pieces of shorthand: the glossary gives the key's meaning for 89, the search passages for 98, both for 70.

Supplied: the shorthand’s form and the key’s meaning in one glossary line, or in one other note the search returned. Both supplied 327 pieces; only the search, 46; only the glossary, 42. Prompts were similar in size: a median of 578 tokens with the search, 621 with the glossary.

02 · The result

Fewer misreadings with the search

The 508 shorthand items scored under No. 13’s rules, each against MaintNorm’s key and then checked blind by the AI readers; shorthand the key leaves as written is not counted. The grey rows are No. 13’s two recorded readings of the same notes on 20 November; the dark rows were run on 21 November, scored after the adjudication.

Shorthand read right, not read, wrong508 items · wrong: search 7, glossary 12
Shorthand read right, not read and read wrong, all three companies, 508 pieces: No help (20 Nov, No. 13): 86% right, 13 not read, 58 wrong; Glossary (20 Nov, No. 13): 97% right, 8 not read, 8 wrong; Glossary (21 Nov, again): 96% right, 8 not read, 12 wrong; Search (21 Nov, other notes): 98% right, 1 not read, 7 wrong.

All three companies, whole notes: meaning kept in 290 of 300 with the search, 273 with the glossary the same day; No. 13 recorded 283 with the glossary and 228 with no help.

03 · Note by note

Five notes, both answers

Each note as typed, what it means by MaintNorm’s key, the first of the other notes the search found for it, and the model’s plain-English reading with each version. Three the search fixed, one of them adding a word from another job; two it broke.

Real work orders · held out from the search and the glossary5 of the 300
  • Company C · as typed

    no a/c fans

    What it means (MaintNorm’s key): no air conditioner fans

    What the search found: the first 4 of 10 other notes
    • no 24volts no a/c = number 24 volts no air conditioner
    • aircon fans not working = air conditioner fans not working
    • replace faulty condenser fans = replace faulty condenser fans
    • Check guards on exhaust fans = check guards on exhaust fans
    With the glossary

    Number air conditioner fans

    ✗ Applies the line “no = number” to “no fans”: the opposite.
    With the search

    No air conditioner fans.

    ✓ Right: other notes read “no a/c” as no air conditioner.
  • Company A · as typed

    XH531 REPLACE CONVERTER SHAFT UNIS

    What it means (MaintNorm’s key): XH531 replace converter shaft universal joints

    What the search found: the first 4 of 10 other notes
    • XH531 REPLACE WIPER ASSY SHAFT & LINKAGE = XH531 replace wiper assembly shaft and linkage
    • Replace trans/converter uni = replace transmission / converter universal joint
    • XH531 Refit Spider bearings to Shaft = XH531 refit spider bearings to shaft
    • XH531 Replace rear drive shaft guard = XH531 replace rear drive shaft guard
    With the glossary

    Asset XH531: Replace converter shaft units.

    ✗ UNIS is not among the 50 lines: read as units.
    With the search

    XH531 replace torque converter shaft universal joints

    ✓ Right on UNIS: another note read “converter uni” as universal joint. Adds “torque”, from another job on the same truck.
  • Company B · as typed

    AL6487 Rotate Pos 3 9 tyres

    What it means (MaintNorm’s key): AL6487 rotate position 3 9 tyres

    What the search found: the first 4 of 10 other notes
    • AL6487 C/O pos 1&2 Tyres = AL6487 change out position 1 and 2 tyres
    • blown pos 9 tyre = blown position 9 tyre
    • FN5736 rotate Pos 3 tyre = FN5736 rotate position 3 tyre
    • AL6487 ROTATE POS 6 & 8 = AL6487 rotate position 6 and 8
    With the glossary

    Asset AL6487: Rotate the 9 tyres at Position 3.

    ✗ Positions 3 and 9 become nine tyres at position 3.
    With the search

    AL6487 rotate position 3 and 9 tyres

    ✓ Right: other notes read “pos 1&2” as positions 1 and 2.
  • Company B · as typed

    LD3192 250hr Service

    What it means (MaintNorm’s key): LD3192 250 hour service

    What the search found: the first 4 of 10 other notes
    • 250H Mech Svce Haul Truck LD3192 = 250 hour mechanical service haul truck LD3192
    • 500H Mech Svce Haul Truck LD3192 = 500 hour mechanical service haul truck LD3192
    • 500H Mech Svce Haul Truck LD3192 = 500 hour mechanical service haul truck LD3192
    • 1000H Mech Svce Haul Truck LD3192 = 1000 hour mechanical service haul truck LD3192
    With the glossary

    Asset LD3192: 250 hour service.

    ✓ Right.
    With the search

    250 hour mechanical service for haul truck LD3192.

    ✗ Adds “mechanical” and “haul truck” from other notes with the same asset number.
  • Company A · as typed

    PH639 REPL POS3 & 3 BRAKE PACKS

    What it means (MaintNorm’s key): PH639 replace position 3 and 3 brake packs

    What the search found: the first 4 of 10 other notes
    • PH639 RETENSION POS 5 & 3 BRAKE PACS = PH639 retension position 5 and 3 brake pads
    • PH639 REPAIR PARK BRAKE FAULT = PH639 repair park brake fault
    • PH639 REPL 6 X BRAKE RELEASE HOSES = PH639 replace 6 x brake release hoses
    • PH639 repair park brake fault = PH639 repair park brake fault
    With the glossary

    Replace position 3 and 3 brake packs on asset PH639.

    ✓ Right.
    With the search

    PH639 replace position 3 and 3 brake pads

    ✗ Copies “brake pads” from another note’s reading of “BRAKE PACS”.

04 · Every note

22 notes only with the search, 5 only with the glossary

One square per note, by whether its meaning came through with each version on the same day. Exact McNemar test, two-sided: p = 0.002. As the readers first judged, blind: 22 against 9, p = 0.03. With the disputed readings counted as lost (four of them kept only by No. 13’s rulings): 286 against 264, 30 against 8.

300 notes · meaning keptsearch 290 · glossary 273
One square per note, 300 notes: meaning kept by both versions 268, only with search 22, only with the glossary 5, by neither 5.

No answer in either version invented a diagnosis or said work was done that the note did not report.

Against No. 13’s glossary run of 20 November the search kept 290 to 283: 13 notes only with the search, 6 only with that run, not a clear difference (p = 0.17). The glossary itself kept 273 on 21 November against 283 on 20 November, with 20 notes changing verdict. On this run the search did better than the glossary; whether it usually does, and by how much, is open.

05 · At your site

Before a search reads your work orders

  1. Index the notes you already haveThe search used the same 3,200 annotated notes the glossary’s 50 lines were counted from, with no lines to choose; in both, the annotation is the work.
  2. Check the readings you indexAnother note read “BRAKE PACS” as brake pads, and the model copied it into a note about brake packs.
  3. Expect borrowed wordsOther notes with the same asset number put “haul truck” into a note that did not say it. Tell the model the other notes are for shorthand only.
  4. Mind what each missesThe glossary and the search each covered about 45 pieces of shorthand the other missed; both together were not tested.
  5. Test on more than one runThe same glossary kept 283 notes one day and 273 the next.

Sources

  1. Bikaun, T., Hodkiewicz, M. and Liu, W. (2024). MaintNorm, a corpus of maintenance short text with each word normalised. Proceedings of W-NUT 2024, pages 68–78. aclanthology.org/2024.wnut-1.7; data github.com/nlp-tlp/maintnorm.
  2. SAG-Wiki, open-source document search, MIT licence; CLI 0.3.11 from npm. github.com/Zleap-AI/SAG-Wiki.
  3. Wu, Y., Liang, X. and Li, G. (2026). SAG2: embedding-free multi-hop retrieval with BM25 and decision models. arXiv:2610.22588 v2. arxiv.org/html/2610.22588v2.
  4. Models: Qwen3-Reranker-0.6B; Qwen 3.8 27B as 4-bit weights.