Expert agreement, repeatability, and accuracy are separate tests

Repeatability measures whether an assessor reproduces a result under specified repeated conditions. Agreement measures whether assessors give matching results. Accuracy compares a result with an adequate reference for the property being determined.

A panel can agree without demonstrating that each member would repeat the answer independently. An assessor can repeat an answer without establishing that it is correct. Combining the words into “expert reliability” does not remove the separate measurements.

Repeated wine samples test individual consistency

At the California State Fair wine competition from 2005 through 2008, judges received repeated samples from the same bottle within a flight. The analysis used their independent scores before panel discussion.

That design tested whether a judge reproduced a score on the same wine. Substantial variation appeared within judges’ repeated scores. Agreement eventually reached in a discussion would answer a different question, because deliberation changes the conditions under which the answer is produced.

Among the small group assessed across two consecutive years, stronger consistency in the first year did not predict stronger consistency in the second. That result limits the use of one year’s performance as evidence of a stable personal trait. It does not establish that no judge can develop repeatable performance under any conditions.

Context resistance requires another test

A separate experiment with wine professionals varied background music. Years of experience did not detectably moderate the context effect on the measured sensory and enjoyment responses.

This finding concerns auditory context in the tested sample. It does not establish that expertise never protects against context, or that results for wine apply to art appraisal or gemstone identification. Experience, repeatability, and resistance to a particular influence remain different attributes.

Gemstone agreement can encounter shared ambiguity

Geographic origin determination compares gemstone observations with reference knowledge. Similar geology in different deposits can produce overlapping characteristics, limiting how decisively those observations distinguish origin.

In the described GIA process, multiple gemologists review a stone and its data, discuss discrepancies, and arrive at a conclusion or an inconclusive determination. Internal consensus is part of that process; it is not a substitute for a uniquely discriminating physical signature.

Take two laboratories that agree on a stone’s origin while relying on overlapping reference characteristics. Their agreement is information about their conclusions. Without an appropriate reference truth and an account of dependence between their methods, it does not produce a numerical error probability simply by counting reports.

Match the test to the intended reliance

For a repeatability question, repeat the assessment under specified conditions. For agreement, compare assessors and preserve whether they deliberated together. For accuracy, identify the reference and the property it establishes.

What varies across domains is the available reference and the measurement process. Wine-judge variation cannot supply a gem laboratory’s error rate. Agreement adds useful information only with its own scope and dependencies retained.