eleticle

Core No. 11Logged 20 Nov 2026Code & data →

Can a local model put the right assay into the right database field?

First test · seven public certificates · 12 fields each · one local model, three attempts · PDF text and rules beside it

We asked a local model to file the copper and gold values on seven public reference-material certificates, with their standard deviations (SD), into twelve lab quality-control database fields. Three passes over the seven with each of two instructions, beside rules written from one certificate, all scored against a key we built by hand.

What we found

  • Define the fields, and it worksWith each field's method, unit and statistic spelled out, the model filed 45 numbers right and left 38 fields rightly empty in each pass over the seven: 83 of 84. Given only the column names, it got 14 to 16 wrong per pass.
  • Rules break on a new layoutWritten from one certificate, the rules got 55 of 84 right, left 20 fields empty that should hold a number, and filed 9 wrong.
  • Right digits, wrong meaningAll 56 wrong entries, from both instructions and the rules, started from a number printed on the certificate, sometimes converted or halved, then filed as the wrong thing: a 2SD as 1SD, ppm as %, indicative as certified.
  • One repeated error with field definitionsOne gold standard's uncertainty, halved, filed as its SD: 0.005 ppm for 0.015. Control limits three times too tight.
  1. Method
  2. 01One field
  3. 02The test
  4. 03The wrong numbers
  5. 04The control chart
  6. 05Before you import

Method

Seven certificates, one model, rules beside it

  • RealSeven public certificatesOREAS 902, 905, 501d and 232b, AMIS0832, CDN-CM-54 and Rocklabs SE147, downloaded on 20 Nov 2026 and read from each PDF's own text layer.
  • TestedOne local model, and rulesQwen 3.8 27B (4-bit) on two 16 GB graphics cards, three attempts per certificate and instruction. Rules written from OREAS 902, then frozen.
  • Checked by handA key, and every missThe key was built before the model's answers were read (OREAS 902 excepted). Every entry the script did not score right was checked by hand.
  • ComputedThe control chartFrom each certificate's numbers. A first test of one model, not a survey.

Fields, instructions, rules, key, scorer and every answer with the model's reasoning: the article's public repository (code MIT, results CC BY 4.0). Its README links each certificate.

01 · One field

Five copper numbers, one field

A reference material's certificate gives each element by several methods. OREAS 902 prints five copper numbers. The site's QAQC database keeps one expected value and one SD per lab method, so each field takes exactly one of them, or none. Pick the field.

OREAS 902 · copper on the certificateFour-acid Cu field ← 0.301 · 1SD 0.008
Five copper numbers on the OREAS 902 certificate on one scale. Highlighted: four-acid digestion, 0.301 wt.%, the one the four-acid field takes.

Certified: 0.111 by 5 % sulphuric acid leach, 0.301 by four-acid digestion, 0.308 by aqua regia, each with its ±2SD bar. Indicative only: 3,120 ppm by laser ablation and 3,887 ppm of CuO by fusion XRF.

The four-acid field takes 0.301 wt.%. Aqua regia's 0.308 and laser ablation's 3,120 ppm sit within 4 % of it, read just as easily, and belong to other methods.

02 · The test

Seven certificates, twelve fields, three readers

Each reader got a certificate's text, as PDF extraction gives it, and filled twelve fields: the value and 1SD for copper by four methods and gold by two. A hand-built key says what each field should hold, often nothing. Pick a reader.

84 fields per attempt · one stripe per attempt83 of 84 right · every attempt
Seven certificates by twelve fields, one stripe per attempt. Every cell is right in all three attempts except SE147's gold 1SD, wrong in all three.

Columns: 4A four-acid, AR aqua regia, CN cyanide, FA fire assay. Model: Qwen 3.8 27B, on this workstation, three attempts per certificate. "Fields defined" names each field's method, unit and statistic and says to file certified values only. "Names only" gives the twelve column names. Rules: written from OREAS 902's layout, then run unchanged.

With every field defined, the model filed 45 of the 46 numbers right in each attempt and kept all 38 empty fields empty. The red stripes are one miss, three times: SE147's 1SD.

03 · The wrong numbers

Read right, filed wrong

Each wrong entry below starts from a number printed on the certificate, read right. What went wrong is which number, and what it stands for.

  1. Model, fields defined · SE147 · 3 of 3

    An uncertainty taken for an SD

    Printed
    Au 0.623 ppm · U ±0.01, k = 2
    Filed
    1SD 0.005 ppm
    Right
    1SD 0.015 ppm, page 4

    U is how well the value is known, not how far results scatter. Limits three times too tight.

  2. Model, names only · CDN-CM-54 · 3 of 3

    A 2SD filed as the 1SD

    Printed
    Copper 0.356 % ± 0.015 · between-lab two SD
    Filed
    1SD 0.015 %
    Right
    1SD 0.0075 %

    Limits twice as wide. Also its other two SDs, and AMIS0832's three.

  3. Model, names only · OREAS 232b · 3 of 3

    ppm filed as %

    Printed
    Cu 26.7 ppm, four-acid
    Filed
    26.7 in the % field
    Right
    0.00267 %

    Ten thousand times too high. The model saw the units differ and kept the number.

  4. Model, names only · OREAS 902 · 3 of 3

    An indicative value filed as certified

    Printed
    Au 46.6 ppb, fire assay, indicative
    Filed
    0.0466 ppm; once 46.6 ppm
    Right
    Empty

    Not certified, no SD. Once also left in ppb: a thousand times too high.

  5. Model, names only · AMIS0832 · 1 of 3

    Acid-soluble filed as cyanide-soluble

    Printed
    Cu AS (acid soluble) 1.57 %
    Filed
    Cyanide-soluble Cu 1.57 %
    Right
    Empty

    Cyanide leach results would be judged against an acid leach.

  6. Rules · OREAS 232b · one run

    Four-acid copper filed under cyanide

    Printed
    4-Acid Digestion · Cu 26.7 ppm
    Filed
    Cyanide-soluble Cu 0.00267 %
    Right
    Four-acid Cu; cyanide empty

    The rules know “4 Acid”, not “4-Acid”: the copper stayed under the last header they knew.

  7. Rules · OREAS 232b and 501d · one run

    A lower limit filed as the SD

    Printed
    Au 0.946 ppm · 95 % uncertainty 0.933–0.959
    Filed
    1SD 0.933 ppm
    Right
    1SD 0.037 ppm

    The second number on this table's row is the lower 95 % limit. Limits 25 times too wide.

04 · The control chart

What a wrong field does

The database's value and SD set the centre and the limits of every QAQC chart for that standard. Here a result outside ±3SD counts as a failure. A field that is too tight flags sound results; one that is too wide passes biased ones.

Results outside ±3SD · by lab biasSE147 gold · 32 % fail at no bias
Share of results failing on SE147 by lab bias. With the certificate's 1SD, 0.3 % at no bias rising to 88 % at 10 % high. With the model's 0.005, 32 % at no bias.

Computed, not measured. Assumed: a lab's own results scatter normally by the certificate's 1SD around its bias, and fail outside ±3SD of the database's numbers, as the OREAS certificates suggest. SE147's 0.015 is the spread of 26 laboratories' averages, not one lab's own scatter, so its curves are an illustration.

Assume a lab's own results scatter normally by 0.015 ppm, the spread the certificate gives between laboratories. Then the model's 0.005 fails 32 % of them at no bias, against 0.3 %: sound results flagged as failures.

It happens

A 2026 technical report on a Zambian copper mine records 16 results on one standard checked against 3.44 % Cu instead of its certified 2.48 %. All 16 failed; with the right value, 12 passed (section 8.5.1.1).

The model's slip is the hard one

The rules' SDs are almost as large as the values, which any range check flags. The model's 0.005 is 0.8 % of the value, inside the 0.6 to 9.9 % of the right SDs in this test: it looks like any other SD.

05 · Before you import

What to check before the numbers reach the database

The model saved most of the typing and read layouts the rules had never seen. These checks catch what it, or anyone, can still get wrong.

  1. Ask for the producer's spreadsheet firstA table needs no reading. OREAS publishes a datapack for each material.In this test: every OREAS certificate names one
  2. Define each field by method, unit and statisticSay “certified only”, and what to do with a 2SD.In this test: 1 wrong per attempt with definitions, 14 to 16 without
  3. Let empty be an answerHere 38 of 84 fields should stay empty.In this test: indicative and uncertified gold filed as certified
  4. Keep the source line with each numberPage, table and row beside each value, for a person to check.In this test: SE147's 0.005 came from “Expanded Uncertainty (U)”
  5. Check the numbers against each otherSoluble below total; each SD a plausible share of its value.In this test: catches the rules' SDs, not SE147's or CDN's
  6. Watch the first results on a new recordIf a third fail at once, suspect the record before the lab.In this test: about a third for SE147's record, at the certificate's scatter