eleticle

Core No. 05Logged 13 Nov 2026Code & data →

How should you clean plant sensor data?

Worked example · real hourly averages · generated records · inserted faults

We generated raw records to match four days of real hourly averages of a mill feed rate, and inserted known faults and plant events. Then we tried common cleaning rules on the records and scored each against the known truth.

What we found

  • One spike hides in an averageOne bad reading moved an hourly average by 137 t/h and left it looking normal. From the average alone, it cannot be repaired.
  • Limits before statisticsA hard upper limit of 1,500 t/h, set from what the instrument can read, caught all fifty spikes in the fifty-spike example. A percentile rule caught none.
  • Odd is not wrongA box-plot rule flagged 787 records of ordinary running, including a real low-feed hour.
  • Purpose decidesThe same start-up is noise for a steady-state model and the whole point of a trip study.
  1. Method
  2. 01Records and averages
  3. 02Negatives
  4. 03Real events
  5. 04What to do
  6. 05Before you clean

Method

Real averages, built records, known faults

  • Real96 hours of feed rateConsecutive hourly averages of mill feed rate from a working plant, renamed and scaled to keep the site anonymous.
  • GeneratedThe records inside each hourBuilt so every untouched hour averages back to its real value within 1 t/h, and thinned the way a historian's exception test thins them.
  • InsertedEvery fault and eventSpikes, negatives, a surge, a trip and a shutdown, at known positions, so each rule could be scored against the truth. Spike sizes are example values.

Averages here are time-weighted, with straight lines between records. A removed record is bridged when the good records either side are no more than 60 seconds apart; a longer gap stays missing and is reported as coverage. A stretch with no records at all is bridged however long, taken as historian compression; on a real tag, check quality flags or an outage log. Your historian may average differently. This shows how each rule behaves on this data, not how often each problem occurs on a real plant.

The records, the code that built and cleaned them, and the figures are in the article's public repository. The code is under the MIT licence and the records under CC BY 4.0.

01 · Records and averages

One shift, as raw records and as hourly averages

Twelve hours of a mill feed rate as a plant historian keeps it: a record every few seconds, between about 850 and 1,120 t/h all shift, except five. Most analysis sees only the hourly averages under the trend, which the historian works out from those records, spikes included. The averages are real; the records and the bad readings were built for the example.

Mill feed rate · t/h · one shift4,571 records · 5 above the instrument span
Top: the raw shift, a flat line with five spikes of up to 1,000,000 t/h. Bottom: the same shift after removing records above 1,500 t/h, moving between about 850 and 1,120 t/h.
HourAverageTrueOff by
11,0491,0490
21,0481,0480
31,0411,0410
41,1871,050+137
51,0451,0450
61,0501,0500
71,0411,042−1
83,2661,048+2,218
9891892−1
101,0411,0410
111,9461,045+901
121,0361,0360
Shift total15,643 t12,387 t+3,256 t

The largest record reads 1,000,000 t/h, so the trend scales to fit it and the real signal becomes a flat line at the bottom. In the averages, two hours are far above anything the weigher can read, and anyone would drop them. Hour 4 reads 1,187 t/h. It is inside the span, it looks like a good hour, and it is 137 t/h too high.

A hard limit

Set by the metallurgist or engineer from what the instrument can report, not from its scaling alone: a valid reading can sit a little past the scaled span, and the transmitter's quality flag says more than the value. Here the span is 0 to 1,500 t/h and the spikes are 100,000 and more. It needs someone who knows the tag, and it keeps working however many bad records there are.

A rule from the data

Ten times the 99th percentile needs no knowledge of the tag, only a choice of percentile and multiplier, and it caught all five spikes. With fifty spikes, just over 1 % of the shift, the 99th percentile is itself a spike. The threshold becomes 1,060,144 t/h and nothing is flagged.

Many spikes are the easy case

With fifty bad records, eleven of the twelve hourly averages are above the span. The problem announces itself. One spike in an hour is harder: nothing marks the average as wrong, and over a shift it adds 136 t that never crossed the belt.

An average cannot be repaired

Once the spike is inside the average, it cannot be taken out again. Clean the raw records first and then average. If averages are all you can get, ask for the maximum, minimum and count beside each one: a maximum of 100,000 gives the hour away.

02 · Negatives

A feed rate cannot be negative. The tag says it is

Two negative readings on the same tag, with two different causes and two different answers.

Mill feed rate · t/h · elapsed hours:minutes
Left: for one hour on an empty running belt the tag reads about minus 8 t/h; cleaning sets it to zero. Right: a failed transmitter reads minus 37.5 t/h for 15 minutes; cleaning removes those records and leaves the stretch missing.

Zero drift: repair it

The belt ran empty for an hour and the weigher read about −8 t/h. The true value is known: zero. Set it to zero and record that you did. Do it only where something other than the reading shows the belt is empty. Here readings down to −15 t/h, 1 % of the span, count as drift, and anything lower as a fault. Left alone, the hour takes 7 t off the total, and the loss grows with every hour the belt runs empty.

A failed transmitter: leave it missing

For fifteen minutes the tag read −37.5 t/h, a fixed low value of the kind a failed transmitter sends, while the plant ran at about 980. The true value is not known. Averaged as it stands, the hour reads 720 t/h: in range, and wrong. Remove the records and report the hour as 972 t/h, the average of the 45 measured minutes, at 75 % coverage. The other 15 minutes stay unknown: do not invent them.

03 · Real events

Some odd values are the plant, not the instrument

Four days of the cleaned tag. It holds a feed surge, a trip, a planned shutdown with its start-up, an hour of empty belt and one hour of low feed. Each is something a plant does, not an instrument fault. Try a statistical rule on them.

Mill feed rate · t/h · four days, cleaned33,701 records
Four days of cleaned feed rate. The box-plot rule keeps only 937 to 1,104 t/h, so it flags the surge, the trip, the shutdown, the empty belt and a real low-feed hour. Three-sigma limits of 732 and 1,289 t/h flag the stops and almost none of the surge.

A 3-sigma rule flags 443 records: most of the stops, and almost none of the surge. A box-plot rule flags 1,605, and 787 of those are ordinary running, including all but one record of the low-feed hour.

The rule cannot tell

Whether a value is a fault or an event is a question about cause. The answer is in the context: the running signal, the maintenance log, the shift report. It is not in the value. The rule also changes its answer with what else is in the data: before the spikes were removed, 3-sigma had limits of about ±20,000 t/h and flagged no event at all.

Label the state, then choose

Mark each record as running, starting or stopped from the plant's own signals, and remove nothing. For a steady-state model, leave out the stops and start-ups by state. For a trip study they are the data. For tonnage they always stay.

04 · What to do

Remove, flag, keep or repair

The answer depends on what the value is and what the data is for.

Kind of valueTraining a steady-state modelInvestigating an eventReporting tonnes
Spike beyond the instrument spanRemoveFlag it may mark the faultRemove
Average that carries a spikeDrop the hourFlag and go to raw recordsRebuild from raw records
Small negative on an empty beltSet to zeroKeep it shows the driftSet to zero
Failed transmitterRemoveKeep it times the failureRemove and show coverage
Start-up, shutdown, tripFilter by stateKeep this is the dataKeep
Surge or low-feed periodKeep it is real variationKeepKeep

05 · Before you clean

Six questions to ask of any tag

  1. What was I given? Raw records, interpolated samples or averages, and which retrieval settings were used.
  2. Have I plotted the raw data? If the trend is a flat line, look for extremes.
  3. Who set the hard limits? Someone who knows the tag: its span and over-range, its quality flags, its units, what is physically possible.
  4. Did I clean before averaging? If only averages exist, ask for maximum, minimum and count as well.
  5. Do I know the plant state? Running, starting or stopped, from more than one signal. Filter by state, not by oddness.
  6. Can I undo it? Flag before deleting, keep the original, and decide by purpose.