eleticle

Core No. 10Logged 20 Nov 2026Code & data →

What does a local LLM know about minerals?

A published test, rebuilt and re-run · two local models · 976 questions × 3 rounds · grading checked by hand

We rebuilt the questions of MineralBench, a 2026 test of what language models know about minerals, from the sources it names, and asked two local models all 976 three times each. A script graded every answer, and we checked its grading by hand.

What we found

  • Words better than factsFlash Next named a geology term from its definition 6 times in 10, but recalled a mineral's basic properties only 4 times in 10.
  • Weakest: numbersSpecific gravity, a mineral's density relative to water: the 27B right 10 % of the time, Flash Next 22 %.
  • Two characters cost up to 16 pointsPicking minerals, rocks and places out of report sentences, told which kinds each holds, the 27B scored 53 and Flash Next 56 out of 100. Without the empty brackets at the end of the paper's prompt: 69 and 68. Missed and extra names both lower the score.
  • Ask again, a different answerFlash Next was right in all three rounds on 157 property questions, and in only one or two on 174 more.
  1. Method
  2. 01The scores
  3. 02The prompt
  4. 03Three rounds
  5. 04Over time
  6. 05On your site

Method

A published test, rebuilt and checked by hand

  • RealMineralBench and its scoresBy Dan and co-authors, a 2026 Research Square preprint (doi:10.21203/rs.3.rs-10717678/v1, CC BY 4.0): its prompts, settings and thirteen models' scores, with each model's release date from its developer.
  • RebuiltThe questionsThe paper's own question files are not public, so the 976 were rebuilt from the sources it names: Mindat's glossary and species list, Wikidata, webmineral and OzROCK. Mindat's definitions and OzROCK's sentences stay out of the repository.
  • RunTwo local modelsQwen 3.8 27B (2 × RTX 5080) and Qwen 3.8 Flash Next (4 × RTX 3090), thinking off, answers as returned on 9 Nov 2026.
  • Checked by handTerm and property answers, plus unmatched namesA script graded each answer. Then all 3,274 distinct term and property answers, and the 484 entities the script did not match, were read beside the key under rules written beforehand: 402 grades changed and 6 entity matches were added. Doubtful keys were kept and logged; one that names no value, a colour given as “Color”, is left out of every score.

The harness, every graded answer, each correction with its reason and the scoring script are in the article's public repository: code under the MIT licence, results under CC BY 4.0.

01 · The scores

They know the words better than the facts

Name the term for a glossary definition, give a mineral's property, pick the entities out of a geology report sentence. Both models did worst on properties.

Score out of 100 · after the hand check975 of 976 questions scored
After the hand check, 27B against Flash Next. Entities in report text, F1: 53 and 56. Share of answers right: term from its definition 51 and 62. A mineral's property: streak 56 and 69, luster 25 and 47, hardness 37 and 47, crystal system 30 and 45, colour 14 and 39, cleavage 26 and 32, element list 19 and 32, specific gravity 10 and 22.

Entities: F1, which weighs entities missed against entities wrongly added, with the paper's own prompt; each sentence came with the kinds of entity in its key. Every other bar: the share of answers right.

Recorded answersElement list · question 3 of 4

AskedRoscoelite: element list?

KeyO, V, Si, K, Al, H

27B
  1. Aluminum, silicon, oxygen, hydrogen
  2. Al, Si, O, H
  3. K, Al
Flash Next
  1. Vanadium, Potassium, Aluminum, Silicon, Oxygen, Hydrogen
  2. Ba, V, K, O
  3. Vanadium, Oxygen, Potassium, Aluminum

Four questions of each kind, picked by a fixed rule that never looked at the answers. Mineral properties are asked as “For the mineral X, what is its property?”; definitions are from Mindat's glossary, sentences from the OzROCK set.

27B got none of its three rounds right; Flash Next got one. The rounds disagree with each other, which is the subject of 03.

02 · The prompt

Two characters in the prompt cost up to 16 points

The entity prompt names the kinds of entity the sentence holds; the paper's ends with Answer:[], an empty list right where the model starts writing. Often the model keeps it empty. Take the brackets out and watch the empty answers go.

Entities in report text · 200 sentencesEnding Answer:[] · the paper's
With the paper's ending, Answer:[]: entity score 27B 53, Flash Next 56; answers read as empty, 27B 26 %, Flash Next 17 %. Without the brackets: 69 and 68, and 5 % and 3 %.

Sentence2 km2 iron ore with limonitic impression and goethitic intercalation; No potential zone 3.

KeyOre deposit: iron ore

27B
  1. nothing
  2. nothing
  3. nothing
Flash Next
  1. nothing
  2. nothing
  3. nothing

With the paper's ending, 26 % of 27B's answers and 17 % of Flash Next's were read as empty, nearly all of them just the empty brackets. Without the brackets, the score rose on 102 of the 200 sentences for 27B and 101 for Flash Next, and fell on 40 and 33.

03 · Three rounds

Ask again, and the answer can change

Every question was asked three times, at the paper's settings, which let a model word its answer differently each time. On a mineral's properties the answers often changed, right and wrong.

575 property questions · rounds rightFlash Next: 157 right every time
The 575 property questions by how many of their three rounds were right. 27B: never 334, once 100, twice 57, all three times 84. Flash Next: never 244, once 102, twice 72, all three times 157.

Flash Next was right in all three rounds on 157 questions and in one or two on 174 more. An answer that is right only some of the time has to be checked every time.

04 · Over time

The paper's best open models rival its closed one

The paper scored thirteen models; only Qwen3-Max keeps its weights closed. Its best open models, out in April 2026, beat it on properties and come within a point of it on entities. The two models here, small enough for a site's own hardware, are shown beside them, not ranked with them: their questions and grading differ.

Score out of 100 · by date first publicA mineral's property · 15 models
A mineral's property, each model by when it first became public and its score out of 100. The paper's closed model Qwen3-Max 65; its best open-weight model on this task, DeepSeek V4 Pro, 81; Qwen 3.5 27B in the paper 21; here, Qwen 3.8 27B 27 and Flash Next 42.

Each dot is a model at the date it first became public, a preview included; the paper does not say which revision of its online models it was served. Its scores come from its own questions and grading; the two here from rebuilt questions, checked by hand (by script alone, 25 and 33 on properties), so they sit beside the paper's, not in its ranking. Terms are left out: scores here ran far above any in the paper.

On a mineral's properties, the paper's best open model, DeepSeek V4 Pro (1.6 trillion parameters), scored 81 and its closed Qwen3-Max 65. The two here scored 27 (27B) and 42 (Flash Next); Qwen 3.5 27B scored 21 in the paper, on different questions and grading.

05 · On your site

Let it read, look the facts up, check a sample

A model of this size, on a site's own hardware, makes a useful first reader. It is not a reference book.

  1. 1 · ReadThe local LLMpulls minerals, rocks and places out of reports and logs.Here: entity F1 69 and 68, prompt fixed, kinds given
  2. 2 · Look upYour own datasupplies the facts: mineral list, lab system, reference table, called as a tool.Here: 27 % and 42 % from memory
  3. 3 · CheckA personreads a sample beside the source, and tries the prompt on your own text first.Here: 402 of 3,274 grades changed

An assistant can use this split: an expert model or a database supplies the values, and the language model reads and explains.