As AI-based face recognition technologies are increasingly adopted for high-stakes applications like locating suspected criminals, public concerns about the accuracy of these technologies have grown as well. These technologies often present a human expert with a shortlist of high-confidence candidate faces from which the expert must select correct match(es) while avoiding false positives, which we term the “last-mile problem.” We propose Second Opinion, a web-based software tool that employs a novel crowdsourcing workflow inspired by cognitive psychology, seed-gather-analyze, to assist experts in solving the last-mile problem. We evaluated Second Opinion with a mixed-methods lab study involving 10 experts and 300 crowd workers who collaborate to identify people in historical photos. We found that crowds can eliminate 75% of false positives from the highest-confidence candidates suggested by face recognition, and that experts were enthusiastic about using Second Opinion in their work. We also discuss broader implications for crowd–AI interaction and crowdsourced person identification.

No abstract

The National Archives contains the Freedmen's Bureau records. Copies of these documents are available through familysearch.org. The documents that I have cited in this paper can be found under FamilySearch's collection labeled "Virginia, Freedmen's Bureau Office Records, 1865-1872." FamilySearch organizes Virginia's records by the Freedmen's Bureau office or subordinate field locations. All of my documents are categorized under Fairfax Courthouse (Fairfax County, assistant subassistant commissioner).