- Maintenance and reliability
- Reading drawings and documents
- Language models
- Open and synthetic datasets
- Evaluation and benchmarks
Does a site glossary help a local LLM read maintenance work orders?
MaintNorm · 300 held-out work orders from three Australian mining companies · Qwen 3.8 27B, with and without each company’s glossary · every answer checked by hand
For each of three Australian mining companies, we built a 50-line glossary of shorthand from 3,200 of its own maintenance work orders. A local model then rewrote 100 other work orders per company, each with some shorthand, in plain English, with and without the glossary, and we checked every answer by hand.
What we found
- 283of 300 work orders kept their meaning with the glossary, against 228 without it
- 58 → 8pieces of shorthand read wrong, of 508, without → with the glossary
- 14 → 0service intervals such as 1000H read as a clock time, without → with the glossary
- 4 → 0answers that invented a diagnosis, without → with. Neither version said work was done that the note did not report
- 4answers where the model applied a glossary line that did not fit
Method
Real work orders, one model, two readings each
- RealMaintNorm's work ordersFrom Bikaun, Hodkiewicz and Liu (W-NUT 2024; MIT licence): 12,000 texts from three Australian mining and mineral processing companies, mostly heavy mobile equipment, each word normalised by one annotator. Asset numbers and names are mock values.
- Kept apartGlossary notes, test notesEach glossary comes from 3,200 of the company's notes; the 100 test notes per company, 300 in all, are a seeded draw from its other notes that hold some shorthand, leaving out any whose text repeats one of its training or validation notes.
- ModelQwen 3.8 27B4-bit weights, on 2 × RTX 5080 (16 GB each) in a workstation, as served: thinking on, its default sampling, one answer per note and version. Median 15 seconds a note with the glossary, 18 without.
- ScoredBy script, then by handEach piece of shorthand against the key, then every answer by hand, blind to the version, under rules written before the run (three diagnosis and five meaning verdicts were corrected afterwards, with the versions known). Synonyms count as right. Each answer was also judged whole: meaning kept, a diagnosis added, work said to be done.
Limits: one model, one answer per note, one annotator’s key (two of its readings look doubtful, logged in the repository), three companies’ notes, not a site’s whole history. MaintNorm has been public since 2024, so the model may have met these notes in training; its effect on this comparison is unknown. 55 test notes match a glossary-building note once asset numbers are set aside (found in No. 16); without them, 230 against 193 of 245 kept their meaning. The glossary version run again a day later kept 273 (No. 16).
The prompt, the glossaries, the 300 notes, every answer with its score and the reason for each hand correction, and the scripts to repeat the run are in the article’s public repository, where the serving software is also named. MaintNorm is at github.com/nlp-tlp/maintnorm; its paper is aclanthology.org/2024.wnut-1.7.
01 · The glossary
Fifty lines from the company’s own notes
MaintNorm holds 4,000 real work orders from each of three Australian mining companies, every word paired with what it means. Each company’s glossary is the 50 shorthand forms used most in 3,200 of its notes, with the annotator’s meaning. The model then read 100 other notes per company, held out from the glossary.
Across all 400 test notes per company. The first few lines do most of the work; past 50, each adds little: about half of what 50 lines miss never appears in the 3,200 glossary-building notes at all.
Same shorthand, another company
cond is conditioner at A and condition at B; h is hour at B, and H-link or hand at C. Eighteen shorthand forms appear in all three glossaries; 19 to 26 are one company’s own.
What counts as shorthand
Every word the key changes: rplc, u/s, 1000H, and misspellings such as drian. Asset numbers are masked in the key and not scored. The 300 notes hold 508 pieces.
02 · The result
The glossary removed the misreadings it covers
Each note was read twice, with the same instruction; the glossary was the only difference. Where it lists the shorthand, wrong readings against MaintNorm’s key went from 47 to none. One more sits on a doubtful key: TEMP REP on a leak, read as temperature repair, as the key also has it.
All three companies: where the glossary lists the shorthand, wrong readings went from 47 to 0; where it does not, from 11 to 8. The meaning of 283 of 300 notes came through with the glossary, 228 without.
03 · Note by note
Six notes, both answers
Each note as typed, what it means by MaintNorm’s key, and the model’s plain-English reading without and with the glossary. Misread words are marked. Four the glossary fixed, one it broke, one it did not cover.
-
Company B · as typed
1000H Mech Svce Wtr Truck XK9760
What it means (MaintNorm’s key): 1000 hour mechanical service water truck XK9760
Without the glossaryAt 10:00, a mechanical service is requested for Water Truck XK9760.
✗ Reads the 1,000-hour service as ten o'clock.With the glossary1000 hour mechanical service water truck XK9760
✓ Right. Glossary line:h = hour -
Company A · as typed
XH531 Replace U/S diff drian plugs
What it means (MaintNorm’s key): XH531 replace unserviceable differential drain plugs
Without the glossaryReplace the upstream differential drain plugs on asset XH531.
✗ Reads U/S as upstream.With the glossaryAsset XH531: Replace the unserviceable differential drain plugs.
✓ Right. Glossary line:u/s = unserviceable -
Company A · as typed
MB119 INV REP ENGINE SHUT DOWN FAULT
What it means (MaintNorm’s key): MB119 investigate repair engine shutdown fault
Without the glossaryRepair the inverter on asset MB119 due to an engine shutdown fault.
✗ Reads INV as inverter and sends the repair there: a faulty inverter the note never named.With the glossaryMB119: Investigate and repair engine shut down fault.
✓ Right. Glossary line:inv = investigate -
Company C · as typed
blinker dipper switch n/w
What it means (MaintNorm’s key): blinker dipper switch not working
Without the glossaryFit a new indicator switch on the excavator.
✗ Reads n/w as new, and a fault report becomes a job.With the glossaryThe blinker on the dipper switch is not working.
✓ Right. Glossary line:n/w = not working -
Company C · as typed
no a/c fans
What it means (MaintNorm’s key): no air conditioner fans
Without the glossaryNo air conditioning fans.
✓ Right.With the glossaryNumber air conditioner fans
✗ Applies the line meant for 'No 2' to 'no fans'. Glossary line:no = number -
Company A · as typed
XH531 REPL REAR ENG F/WHEEL HOSU SEE TEX
What it means (MaintNorm’s key): XH531 replace rear engine flywheel housing see text
Without the glossaryAsset XH531: Replace the rear engine front wheel HOSU. See text.
✗ F/WHEEL is the flywheel; HOSU, its housing, is left unread.With the glossaryReplace the rear engine front wheel hose. See TEX.
✗ Neither is in the glossary: the same misreading, and a guess.
04 · Faithful to the note
No repair claimed done, four diagnoses invented
Every answer was read by hand, without knowing which had the glossary: does it say what the note says? Shorthand left visibly unread counts as kept; a wrong reading does not. Each square is one note, in the same place in both grids.
Of the 17 misses with the glossary, six repeat an error made without it, four come from a glossary line that did not fit, three are shorthand it does not list, two are plain words misread, one drops the asset number and one rests on a doubtful key.
Three lines of the instruction, the same in both
- Keep to what the note says. Do not add a cause, a diagnosis, a fault or an outcome that the note does not state.
- Keep the note’s tense. A request for work stays a request: say that work was done only if the note says so.
- If you cannot read a piece of shorthand with confidence, keep it as written … and list it under “unsure”.
Neither version claimed a completed repair, but misreadings remained, and all four invented diagnoses grew from them, without the glossary: INV read as inverter three times (“repair the inverter due to an engine shutdown fault”), and “No power in cab - reset” sent to a circuit breaker.
05 · At your site
Before a model reads your work orders
- Build the glossary from your own notesCount the shorthand in a year of work orders and have a planner check the top lines. Fifty covered 59 to 82% of the shorthand here.
- Write each line the way it will be readThe model took “quick = Kwik” and “no = number” literally: two quick fill connectors became Kwik, and “no a/c fans” became “Number air conditioner fans”. Write “quick tip = Kwik Tip” and “No 2 = number 2”.
- Spell out units and intervalsWithout “h = hour”, service intervals such as 1000H became clock times in 14 answers.
- Check what the glossary does not coverWith the glossary, the shorthand still misread lay outside its 50 lines, one doubtful case aside: F/WHEEL, ext, INJ.
- Keep the note beside the readingThe reading helps someone who does not know the shorthand. The note stays the record.