Identify Experts through Revealed Confidence: Application to Wisdom of Crowds

Identifying the experts among the crowds may help us further improve the wisdom of crowds. However, due to overconfidence, conventional confidence elicitation can't reliably identify experts. To better reflect the relative expertise among agents, we propose the Revealed Confidence (RC) algorithm, which uses belief-update under an influence as a measure for confidence. We highlight the distinction between first-order and second-order uncertainty. We demonstrate that neglecting second-order uncertainty, as in the case of conventional self-reported confidence, may lead a perfectly rational agent to exhibit over-precision. From our belief-updating model, we prove that RC reveals both first-order and second-order uncertainty, as well as clarify the implications of the distance effect (the distance between an initial answer and its influence affects belief-updating) to the revelation of uncertainty. Our empirical analysis test, and confirm, several of our model predictions that (1) RC is able to improve upon conventional wisdom of crowds methods by overweighting the more accurate agents (2) the distance effect indeed affects belief-updating henceforth RC algorithm's performance and should be carefully controlled for. We also compare RC algorithm with the state-of-the-art minimal Pivoting Method and show RC still has an edge.