A generative Bayesian model for aggregating experts' probabilities
UAI '04 Proceedings of the 20th conference on Uncertainty in artificial intelligence
Neighborhood-Based Local Sensitivity
ECML '07 Proceedings of the 18th European conference on Machine Learning
| Hi-index | 0.00 |
In order to improve forecasts, a decision-maker often combines probabilities given by various sources, such as human experts and machine-learning classifiers. When few training data are available, aggregation can be improved by incorporating prior knowledge about both the event being forecasted and salient properties of the experts. To this end, we develop a generative Bayesian aggregation model for probabilistic classification of both binomial and multinomial events. The model includes an event-specific prior distribution, measures of individual experts' bias, calibration, and accuracy, and a measure of dependence between experts. Rather than require absolute measures, we show that the aggregate forecast may be expressed in terms of relative accuracy between experts. The model results in a weighted logarithmic opinion pool (LogOps) that satisfies the external Bayesian property, as well as a related consistency property that we term “invariance to indistinguishable outcomes.” We derive analytic solutions for the special cases of independent experts and for exchangeable experts. The model's application to decision analysis is demonstrated in a betting problem, calculating the value of experts as a function of their accuracy. We develop methods for assessing the model's parameters, as well as for learning them from data. Testing the model on simulated and real-world data demonstrates the model's accuracy to be comparable to or better than other aggregation methods.