topic Re: What do you do when you suspect that some data is wrong? in Mathcad

What do you do when you suspect that some data is wrong?

Fred_Kohlhepp — Thu, 10 Aug 2023 12:14:05 GMT

I spent a fair portion of my career taking measurements of experimental data. When I first started the senior engineer would review the data and discard measurements that he deemed false. Discarding data simply because you thought it was wrong struck me as very questionable and I spent a fair amount of effort developing a method (using T statistics) to identify "bad" data.

It turns out that there is a statistically rigorous calculation to do just that, developed (and published in 1852) by Benjamin Peirce. That method is developed and demonstrated in the attached Prime 4 Express file.

Thoughts and suggestions?

Re: What do you do when you suspect that some data is wrong?

Derbigdog — Thu, 10 Aug 2023 17:40:44 GMT

W. Edwards Deming wrote a book on Statistical Process Control that addressed this subject as well. His ideas lead to the Japanese manufacturing industry going from one of the worst to one of the best in quality control.

Re: What do you do when you suspect that some data is wrong?

FDS — Fri, 11 Aug 2023 12:19:32 GMT

Dear Fred, Would it be possible to attach a pdf of your worksheet? Have a nice weekend.

Re: What do you do when you suspect that some data is wrong?

Werner_E — Fri, 11 Aug 2023 12:32:48 GMT

Would your method also work in case of the data provided in this thread?
Solved: Remove specific regions from a graph - PTC Community

The built-in functions (Grubbs, GrubbsClassic, ThreeSigma) won't.

Re: What do you do when you suspect that some data is wrong?

Fred_Kohlhepp — Fri, 11 Aug 2023 14:39:02 GMT

Per your request.

Re: What do you do when you suspect that some data is wrong?

Fred_Kohlhepp — Fri, 11 Aug 2023 15:43:35 GMT

Man! That's a lot of data!

First, note that Terry has successfully trimmed this.

Second, Peirce (takes the log of the inequality I'm using. I haven't been able to do that successfully. The large data set is(I think) keeping my solution from working (using the root function.)

Third, my first pass simply treated the entire set as measurements to be averaged and analyzed. Clearly there's a sinusoidal function that might reduce scatter and standard deviation.

Re: What do you do when you suspect that some data is wrong?

Werner_E — Fri, 11 Aug 2023 16:34:21 GMT

@Fred_Kohlhepp wrote:

Man! That's a lot of data!

First, note that Terry has successfully trimmed this.

Second, Peirce (takes the log of the inequality I'm using. I haven't been able to do that successfully. The large data set is(I think) keeping my solution from working (using the root function.)

Third, my first pass simply treated the entire set as measurements to be averaged and analyzed. Clearly there's a sinusoidal function that might reduce scatter and standard deviation.

Yes, sure a huge amount of data - slightly less than 175000 data points.
As far as I understood Terry guessed(!) a sine as being the upper limit and eliminated all data above it.

I thought about automating that process by using an outlier function (without success).

If we zoom in (see picture below) we can clearly(?) see which data should be considered an outlier/spike. I thought about some kind of windowing and applying an outlier function peu à peu instead of treating all the data in one go ....?
I had not played around with that idea any further as the OP in that thread seemed to be happy with Terrys solution anyway.

The thread just came to my mind when I read your posting.

Re: What do you do when you suspect that some data is wrong?

Fred_Kohlhepp — Sat, 12 Aug 2023 12:23:00 GMT

I spent some more time to figure out why this system wasn't working.

The attacked file points out a major flaw: The procedure (as presented here) will not work if the data set is larger than 142 points!

Back to the drawing board!!

Re: What do you do when you suspect that some data is wrong?

FDS — Sat, 12 Aug 2023 18:37:20 GMT

Dear Fred, Thank you highly appreciated!

Re: What do you do when you suspect that some data is wrong?

Fred_Kohlhepp — Fri, 25 Aug 2023 16:20:00 GMT

Working to expand sample size limit, got the logarithmic equation to work. Not sure why there're still issues.

Still Prime 4 Express