- Laboratory and assay
- Reading drawings and documents
- Language models
- Assays and lab data
- Documents and drawings
Can a local model put the right assay into the right database field?
First test · seven public certificates · 12 fields each · one local model, three attempts · PDF text and rules beside it
We asked a local model to file the copper and gold values on seven public reference-material certificates, with their standard deviations (SD), into twelve lab quality-control database fields. Three passes over the seven with each of two instructions, beside rules written from one certificate, all scored against a key we built by hand.
What we found
- Define the fields, and it worksWith each field's method, unit and statistic spelled out, the model filed 45 numbers right and left 38 fields rightly empty in each pass over the seven: 83 of 84. Given only the column names, it got 14 to 16 wrong per pass.
- Rules break on a new layoutWritten from one certificate, the rules got 55 of 84 right, left 20 fields empty that should hold a number, and filed 9 wrong.
- Right digits, wrong meaningAll 56 wrong entries, from both instructions and the rules, started from a number printed on the certificate, sometimes converted or halved, then filed as the wrong thing: a 2SD as 1SD, ppm as %, indicative as certified.
- One repeated error with field definitionsOne gold standard's uncertainty, halved, filed as its SD: 0.005 ppm for 0.015. Control limits three times too tight.
Method
Seven certificates, one model, rules beside it
- RealSeven public certificatesOREAS 902, 905, 501d and 232b, AMIS0832, CDN-CM-54 and Rocklabs SE147, downloaded on 20 Nov 2026 and read from each PDF's own text layer.
- TestedOne local model, and rulesQwen 3.8 27B (4-bit) on two 16 GB graphics cards, three attempts per certificate and instruction. Rules written from OREAS 902, then frozen.
- Checked by handA key, and every missThe key was built before the model's answers were read (OREAS 902 excepted). Every entry the script did not score right was checked by hand.
- ComputedThe control chartFrom each certificate's numbers. A first test of one model, not a survey.
Fields, instructions, rules, key, scorer and every answer with the model's reasoning: the article's public repository (code MIT, results CC BY 4.0). Its README links each certificate.
01 · One field
Five copper numbers, one field
A reference material's certificate gives each element by several methods. OREAS 902 prints five copper numbers. The site's QAQC database keeps one expected value and one SD per lab method, so each field takes exactly one of them, or none. Pick the field.
Certified: 0.111 by 5 % sulphuric acid leach, 0.301 by four-acid digestion, 0.308 by aqua regia, each with its ±2SD bar. Indicative only: 3,120 ppm by laser ablation and 3,887 ppm of CuO by fusion XRF.
The four-acid field takes 0.301 wt.%. Aqua regia's 0.308 and laser ablation's 3,120 ppm sit within 4 % of it, read just as easily, and belong to other methods.
02 · The test
Seven certificates, twelve fields, three readers
Each reader got a certificate's text, as PDF extraction gives it, and filled twelve fields: the value and 1SD for copper by four methods and gold by two. A hand-built key says what each field should hold, often nothing. Pick a reader.
Columns: 4A four-acid, AR aqua regia, CN cyanide, FA fire assay. Model: Qwen 3.8 27B, on this workstation, three attempts per certificate. "Fields defined" names each field's method, unit and statistic and says to file certified values only. "Names only" gives the twelve column names. Rules: written from OREAS 902's layout, then run unchanged.
With every field defined, the model filed 45 of the 46 numbers right in each attempt and kept all 38 empty fields empty. The red stripes are one miss, three times: SE147's 1SD.
03 · The wrong numbers
Read right, filed wrong
Each wrong entry below starts from a number printed on the certificate, read right. What went wrong is which number, and what it stands for.
Model, fields defined · SE147 · 3 of 3
An uncertainty taken for an SD
- Printed
- Au 0.623 ppm · U ±0.01, k = 2
- Filed
- 1SD 0.005 ppm
- Right
- 1SD 0.015 ppm, page 4
U is how well the value is known, not how far results scatter. Limits three times too tight.
Model, names only · CDN-CM-54 · 3 of 3
A 2SD filed as the 1SD
- Printed
- Copper 0.356 % ± 0.015 · between-lab two SD
- Filed
- 1SD 0.015 %
- Right
- 1SD 0.0075 %
Limits twice as wide. Also its other two SDs, and AMIS0832's three.
Model, names only · OREAS 232b · 3 of 3
ppm filed as %
- Printed
- Cu 26.7 ppm, four-acid
- Filed
- 26.7 in the % field
- Right
- 0.00267 %
Ten thousand times too high. The model saw the units differ and kept the number.
Model, names only · OREAS 902 · 3 of 3
An indicative value filed as certified
- Printed
- Au 46.6 ppb, fire assay, indicative
- Filed
- 0.0466 ppm; once 46.6 ppm
- Right
- Empty
Not certified, no SD. Once also left in ppb: a thousand times too high.
Model, names only · AMIS0832 · 1 of 3
Acid-soluble filed as cyanide-soluble
- Printed
- Cu AS (acid soluble) 1.57 %
- Filed
- Cyanide-soluble Cu 1.57 %
- Right
- Empty
Cyanide leach results would be judged against an acid leach.
Rules · OREAS 232b · one run
Four-acid copper filed under cyanide
- Printed
- 4-Acid Digestion · Cu 26.7 ppm
- Filed
- Cyanide-soluble Cu 0.00267 %
- Right
- Four-acid Cu; cyanide empty
The rules know “4 Acid”, not “4-Acid”: the copper stayed under the last header they knew.
Rules · OREAS 232b and 501d · one run
A lower limit filed as the SD
- Printed
- Au 0.946 ppm · 95 % uncertainty 0.933–0.959
- Filed
- 1SD 0.933 ppm
- Right
- 1SD 0.037 ppm
The second number on this table's row is the lower 95 % limit. Limits 25 times too wide.
04 · The control chart
What a wrong field does
The database's value and SD set the centre and the limits of every QAQC chart for that standard. Here a result outside ±3SD counts as a failure. A field that is too tight flags sound results; one that is too wide passes biased ones.
Computed, not measured. Assumed: a lab's own results scatter normally by the certificate's 1SD around its bias, and fail outside ±3SD of the database's numbers, as the OREAS certificates suggest. SE147's 0.015 is the spread of 26 laboratories' averages, not one lab's own scatter, so its curves are an illustration.
Assume a lab's own results scatter normally by 0.015 ppm, the spread the certificate gives between laboratories. Then the model's 0.005 fails 32 % of them at no bias, against 0.3 %: sound results flagged as failures.
It happens
A 2026 technical report on a Zambian copper mine records 16 results on one standard checked against 3.44 % Cu instead of its certified 2.48 %. All 16 failed; with the right value, 12 passed (section 8.5.1.1).
The model's slip is the hard one
The rules' SDs are almost as large as the values, which any range check flags. The model's 0.005 is 0.8 % of the value, inside the 0.6 to 9.9 % of the right SDs in this test: it looks like any other SD.
05 · Before you import
What to check before the numbers reach the database
The model saved most of the typing and read layouts the rules had never seen. These checks catch what it, or anyone, can still get wrong.
- Ask for the producer's spreadsheet firstA table needs no reading. OREAS publishes a datapack for each material.In this test: every OREAS certificate names one
- Define each field by method, unit and statisticSay “certified only”, and what to do with a 2SD.In this test: 1 wrong per attempt with definitions, 14 to 16 without
- Let empty be an answerHere 38 of 84 fields should stay empty.In this test: indicative and uncertified gold filed as certified
- Keep the source line with each numberPage, table and row beside each value, for a person to check.In this test: SE147's 0.005 came from “Expanded Uncertainty (U)”
- Check the numbers against each otherSoluble below total; each SD a plausible share of its value.In this test: catches the rules' SDs, not SE147's or CDN's
- Watch the first results on a new recordIf a third fail at once, suspect the record before the lab.In this test: about a third for SE147's record, at the certificate's scatter