10

Lets say you have a measurement device of which you do not know the accuracy, and a reference measurement device. Both measure a variable $x$. The range of interest is $x_0<x < x_1$. How do you determine the accuracy of the unknown device in this range?

My course of action would be to gather values for both devices from $x_0$ to $x_1$ and building a distribution of errors. The accuracy could then be the error span, $\pm3\sigma$ or something similar - is this correct?

Assumptions:

  • The reference measurement device is calibrated and has virtually no error
JHK
  • 1,638
  • 2
  • 18
  • 29
  • First the measurement reference device should be properly calibrated. Then the accuracy can be determined by the minimum input for witch there is change in the output. Ex. in vernier callipers the minimum input is the distance. – Fennekin Jan 18 '16 at 15:26
  • I added the assumption that it is calibrated. - I dont really know how to apply your example with vernier callipers – JHK Jan 18 '16 at 18:54
  • 1
    ok, so 'accuracy' or 'trueness' per your reference? It looks like you'd first determine trueness of the reference device, assuming precision as stated for the device, then compare your mean reading on the test device to determine trueness, and calculate the variance to determine precision. I think most of us "old-timers" use "accuracy" for this newfangled "trueness." – Carl Witthoft Jan 18 '16 at 20:47

3 Answers3

3

The only way to determine the accuracy to which any measuring device provides measurements is to calibrate it against a device of known accuracy and known errors for measurements.

You technique is partially correct; don't just do the error measurement for the limits of the device as one population or sample bin. This is because measurement errors are not always uniform.

For example, for readings between 0 & 1, the error might be -0.2 and for readings between 2 & 3 the error might be +0.6. Your testing must be done in ranges or bands, irrespective of whether the units are mm (for rulers), m/s (for anemometers or speedometers) or Pa (for barometers).

For each range/band you determine the error for that range/band and then apply that error to the measurement taken from the device that needed calibrating.

Fred
  • 9,562
  • 11
  • 32
  • 44
  • That would be with the assumption that I want to calibrate the device - which I did not want to. I just want to evaluate it. I understand your example, and to go a step further with it: You now have -0.2 error for the first "bin", +0.6 error for the second and so on ... If you want to summarize that for the entire range in a single number, what do you use? Standard deviation for all errors? – JHK Jan 19 '16 at 10:05
  • How could you "evaluate" without simultaneously generating info which provides calibration? – Carl Witthoft Jan 19 '16 at 12:24
  • @CarlWitthoft Well, you generate info for calibration, but it may not be possible to embed that info into the device or making it available in other form for the user of it. – JHK Jan 19 '16 at 12:51
  • @CarlWitthoft http://www.thefreedictionary.com/calibrate disagrees with me. I still think there's a difference between determining an accuracy and improving a device by making that information available, but I dont think we're making progress by arguing about this definition. – JHK Jan 19 '16 at 12:55
  • 1
    "The only way to determine the accuracy ... is to calibrate it against a device of known accuracy" isn't true. If it were we would never be able to calibrate our most accurate devices. To determine the precision of instruments for which there isn't a more accurate device you compare it to identical copies of itself (or use physical reasoning). – Chris Mueller Jan 19 '16 at 12:57
2

Your approach is broadly correct.

If you are only interested in the accuracy of your system you probably want to use something like the maximum error. Your accuracy is then +/- Max error with the assumption that real errors are uniformly distributed within this range (a uniform distribution will often be an overestimation but is a simple option when no better information is available).

However, this approach will often produce large errors due to systematic effects which can easily be corrected by fitting a curve (normally linear) through the plot of measured and true values.

This should correct for the bias in your instrument and you can then calculate the uncertainty based on the standard deviation of the residuals. The total uncertainty is normally a multiple of $\sigma$, the choice is fairly arbitary, so you should state the multiple (k value), or the associate coverage factor.. You should also state what distribution you are assuming as this will effect what multiple gives a specific coverage. E.g. For a Gaussian 95 % coverage k~2, but for a uniform distribution 95 % coverage k~1.68

nivag
  • 1,025
  • 8
  • 14
  • As you said, the choice is arbitrary, but is there a common standard? I have often encountered data sheets which just said "Accuracy: +-Y", without defining if this is 2sigma, 3sigma, etc. ... – JHK Jan 19 '16 at 16:44
  • @JohnH.K. 2$\sigma$ is normally the default or 95 % confidence interval. Not stating what it is not good practice though. Although if it just says accuracy I would think/hope it is talking about a max permissible error. – nivag Jan 20 '16 at 16:32
  • In my experience data sheets are often a mess of unclear terms and badly defined values though (sales brouchers are even worse). – nivag Jan 20 '16 at 16:34
1

I was on a team of quality engineers (but not one of the experts), and they had a visual where they used a 2d plot where X axis was first measurement and Y was second measurement of the same observable feature.

They would repeat the measure/remeasure and create what they called a "sausage chart". They would eliminate the outlying 2% of samples and draw a "sausage" around the rest.

You could visually see the quality of the measurement system by observing how close the data points fell to the 45deg angle line.

Baronz
  • 113
  • 6