eleticle

Core No. 13Logged 20 Nov 2026Code & data →

Does a site glossary help a local LLM read maintenance work orders?

MaintNorm · 300 held-out work orders from three Australian mining companies · Qwen 3.8 27B, with and without each company’s glossary · every answer checked by hand

For each of three Australian mining companies, we built a 50-line glossary of shorthand from 3,200 of its own maintenance work orders. A local model then rewrote 100 other work orders per company, each with some shorthand, in plain English, with and without the glossary, and we checked every answer by hand.

What we found

  • 283of 300 work orders kept their meaning with the glossary, against 228 without it
  • 58 → 8pieces of shorthand read wrong, of 508, without → with the glossary
  • 14 → 0service intervals such as 1000H read as a clock time, without → with the glossary
  • 4 → 0answers that invented a diagnosis, without → with. Neither version said work was done that the note did not report
  • 4answers where the model applied a glossary line that did not fit
  1. Method
  2. 01The glossary
  3. 02The result
  4. 03Note by note
  5. 04Faithful to the note
  6. 05At your site

Method

Real work orders, one model, two readings each

  • RealMaintNorm's work ordersFrom Bikaun, Hodkiewicz and Liu (W-NUT 2024; MIT licence): 12,000 texts from three Australian mining and mineral processing companies, mostly heavy mobile equipment, each word normalised by one annotator. Asset numbers and names are mock values.
  • Kept apartGlossary notes, test notesEach glossary comes from 3,200 of the company's notes; the 100 test notes per company, 300 in all, are a seeded draw from its other notes that hold some shorthand, leaving out any whose text repeats one of its training or validation notes.
  • ModelQwen 3.8 27B4-bit weights, on 2 × RTX 5080 (16 GB each) in a workstation, as served: thinking on, its default sampling, one answer per note and version. Median 15 seconds a note with the glossary, 18 without.
  • ScoredBy script, then by handEach piece of shorthand against the key, then every answer by hand, blind to the version, under rules written before the run (three diagnosis and five meaning verdicts were corrected afterwards, with the versions known). Synonyms count as right. Each answer was also judged whole: meaning kept, a diagnosis added, work said to be done.

Limits: one model, one answer per note, one annotator’s key (two of its readings look doubtful, logged in the repository), three companies’ notes, not a site’s whole history. MaintNorm has been public since 2024, so the model may have met these notes in training; its effect on this comparison is unknown. 55 test notes match a glossary-building note once asset numbers are set aside (found in No. 16); without them, 230 against 193 of 245 kept their meaning. The glossary version run again a day later kept 273 (No. 16).

The prompt, the glossaries, the 300 notes, every answer with its score and the reason for each hand correction, and the scripts to repeat the run are in the article’s public repository, where the serving software is also named. MaintNorm is at github.com/nlp-tlp/maintnorm; its paper is aclanthology.org/2024.wnut-1.7.

01 · The glossary

Fifty lines from the company’s own notes

MaintNorm holds 4,000 real work orders from each of three Australian mining companies, every word paired with what it means. Each company’s glossary is the 50 shorthand forms used most in 3,200 of its notes, with the annotator’s meaning. The model then read 100 other notes per company, held out from the glossary.

Shorthand in new notes the glossary covers50 lines: A 72% · B 82% · C 59%
For each company, the share of the shorthand in its 400 test notes that a glossary of its most used shorthand forms covers, against the glossary's length. Fifty lines cover 72 % for company A, 82 % for B and 59 % for C. At 150 lines: 82 %, 88 % and 73 %.

Across all 400 test notes per company. The first few lines do most of the work; past 50, each adds little: about half of what 50 lines miss never appears in the 3,200 glossary-building notes at all.

Same shorthand, another company

cond is conditioner at A and condition at B; h is hour at B, and H-link or hand at C. Eighteen shorthand forms appear in all three glossaries; 19 to 26 are one company’s own.

What counts as shorthand

Every word the key changes: rplc, u/s, 1000H, and misspellings such as drian. Asset numbers are masked in the key and not scored. The 300 notes hold 508 pieces.

02 · The result

The glossary removed the misreadings it covers

Each note was read twice, with the same instruction; the glossary was the only difference. Where it lists the shorthand, wrong readings against MaintNorm’s key went from 47 to none. One more sits on a doubtful key: TEMP REP on a leak, read as temperature repair, as the key also has it.

Shorthand read right, not read, read wrong508 pieces · wrong 58 → 8
Shorthand read right, not read and read wrong, all three companies. Listed in the glossary: wrong readings 47 without it, 0 with it. Not listed: 11 and 8.

All three companies: where the glossary lists the shorthand, wrong readings went from 47 to 0; where it does not, from 11 to 8. The meaning of 283 of 300 notes came through with the glossary, 228 without.

03 · Note by note

Six notes, both answers

Each note as typed, what it means by MaintNorm’s key, and the model’s plain-English reading without and with the glossary. Misread words are marked. Four the glossary fixed, one it broke, one it did not cover.

Real work orders · held out from the glossary6 of the 300
  • Company B · as typed

    1000H Mech Svce Wtr Truck XK9760

    What it means (MaintNorm’s key): 1000 hour mechanical service water truck XK9760

    Without the glossary

    At 10:00, a mechanical service is requested for Water Truck XK9760.

    ✗ Reads the 1,000-hour service as ten o'clock.
    With the glossary

    1000 hour mechanical service water truck XK9760

    ✓ Right. Glossary line: h = hour
  • Company A · as typed

    XH531 Replace U/S diff drian plugs

    What it means (MaintNorm’s key): XH531 replace unserviceable differential drain plugs

    Without the glossary

    Replace the upstream differential drain plugs on asset XH531.

    ✗ Reads U/S as upstream.
    With the glossary

    Asset XH531: Replace the unserviceable differential drain plugs.

    ✓ Right. Glossary line: u/s = unserviceable
  • Company A · as typed

    MB119 INV REP ENGINE SHUT DOWN FAULT

    What it means (MaintNorm’s key): MB119 investigate repair engine shutdown fault

    Without the glossary

    Repair the inverter on asset MB119 due to an engine shutdown fault.

    ✗ Reads INV as inverter and sends the repair there: a faulty inverter the note never named.
    With the glossary

    MB119: Investigate and repair engine shut down fault.

    ✓ Right. Glossary line: inv = investigate
  • Company C · as typed

    blinker dipper switch n/w

    What it means (MaintNorm’s key): blinker dipper switch not working

    Without the glossary

    Fit a new indicator switch on the excavator.

    ✗ Reads n/w as new, and a fault report becomes a job.
    With the glossary

    The blinker on the dipper switch is not working.

    ✓ Right. Glossary line: n/w = not working
  • Company C · as typed

    no a/c fans

    What it means (MaintNorm’s key): no air conditioner fans

    Without the glossary

    No air conditioning fans.

    ✓ Right.
    With the glossary

    Number air conditioner fans

    ✗ Applies the line meant for 'No 2' to 'no fans'. Glossary line: no = number
  • Company A · as typed

    XH531 REPL REAR ENG F/WHEEL HOSU SEE TEX

    What it means (MaintNorm’s key): XH531 replace rear engine flywheel housing see text

    Without the glossary

    Asset XH531: Replace the rear engine front wheel HOSU. See text.

    ✗ F/WHEEL is the flywheel; HOSU, its housing, is left unread.
    With the glossary

    Replace the rear engine front wheel hose. See TEX.

    ✗ Neither is in the glossary: the same misreading, and a guess.

04 · Faithful to the note

No repair claimed done, four diagnoses invented

Every answer was read by hand, without knowing which had the glossary: does it say what the note says? Shorthand left visibly unread counts as kept; a wrong reading does not. Each square is one note, in the same place in both grids.

300 notes · each answermeaning kept 228 → 283
Two grids of 300 squares, one per note. Without the glossary, 72 answers misread or change the note, four of them by adding a diagnosis. With the glossary, 17 misread or change it. No answer in either says the work was done.

Of the 17 misses with the glossary, six repeat an error made without it, four come from a glossary line that did not fit, three are shorthand it does not list, two are plain words misread, one drops the asset number and one rests on a doubtful key.

Three lines of the instruction, the same in both

  • Keep to what the note says. Do not add a cause, a diagnosis, a fault or an outcome that the note does not state.
  • Keep the note’s tense. A request for work stays a request: say that work was done only if the note says so.
  • If you cannot read a piece of shorthand with confidence, keep it as written … and list it under “unsure”.

Neither version claimed a completed repair, but misreadings remained, and all four invented diagnoses grew from them, without the glossary: INV read as inverter three times (“repair the inverter due to an engine shutdown fault”), and “No power in cab - reset” sent to a circuit breaker.

05 · At your site

Before a model reads your work orders

  1. Build the glossary from your own notesCount the shorthand in a year of work orders and have a planner check the top lines. Fifty covered 59 to 82% of the shorthand here.
  2. Write each line the way it will be readThe model took “quick = Kwik” and “no = number” literally: two quick fill connectors became Kwik, and “no a/c fans” became “Number air conditioner fans”. Write “quick tip = Kwik Tip” and “No 2 = number 2”.
  3. Spell out units and intervalsWithout “h = hour”, service intervals such as 1000H became clock times in 14 answers.
  4. Check what the glossary does not coverWith the glossary, the shorthand still misread lay outside its 50 lines, one doubtful case aside: F/WHEEL, ext, INJ.
  5. Keep the note beside the readingThe reading helps someone who does not know the shorthand. The note stays the record.