The Multidimensional Wisdom of Crowds
Peter Welinder1 Steve Branson2 Serge Belongie2 Pietro Perona1
1 California Institute of Technology, 2 University of California, San Diego
{welinder,perona}@caltech.edu {sbranson,sjb}@cs.ucsd.edu
Abstract
Distributing labeling tasks among hundreds or thousands of annotators is an in-
creasingly important method for annotating large datasets. We present a method
for estimating the underlying value (e.g. the class) of each image from (noisy) an-
notations provided by multiple annotators. Our method is based on a model of the
image formation and annotation process. Each image has different characteristics
that are represented in an abstract Euclidean space. Each annotator is modeled as
a multidimensional entity with variables representing competence, expertise and
bias. This allows the model to discover and represent groups of annotators that
have different sets of skills and knowledge, as well as groups of images that differ
qualitativel
image/model/notators/groups/California/tivel/机器/methodfor/present/class/
image/model/notators/groups/California/tivel/机器/methodfor/present/class/
-->