Best of both worlds: Human-machine collaboration for object annotation

Russakovsky, Olga; Li, Lijia; Li, Feifei

doi:10.1109/cvpr.2015.7298824

Cited by 193 publications

(132 citation statements)

References 54 publications

(116 reference statements)

Supporting

Mentioning

129

Contrasting

Unclassified

Order By: Relevance

“…many ML-methods perform very badly on extrapolation problems which would be very easy for humans [32, p. 4] -and major assumptions of normative models cannot be applied in reality, a conjoint approach of human and machine input could be key to enhanced decision quality. Therefore, the answer is to put humans in the loop [40]. However, using normative models to integrate human decision making in centrals parts of machine learning could lead to faulty predictions since the nature of actual decision making is of bounded rationality [5].…”

Section: Open Problemsmentioning

confidence: 99%

See 1 more Smart Citation

Reasoning Under Uncertainty: Towards Collaborative Interactive Machine Learning

Robert

Büttner

Röcker

et al. 2016

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. In this paper, we present the current state-of-the-art of decision making (DM) and machine learning (ML) and bridge the two research domains to create an integrated approach of complex problem solving based on human and computational agents. We present a novel classification of ML, emphasizing the human-in-the-loop in interactive ML (iML) and more specific on collaborative interactive ML (ciML), which we understand as a deep integrated version of iML, where humans and algorithms work hand in hand to solve complex problems. Both humans and computers have specific strengths and weaknesses and integrating humans into machine learning processes might be a very efficient way for tackling problems. This approach bears immense research potential for various domains, e.g., in health informatics or in industrial applications. We outline open questions and name future challenges that have to be addressed by the research community to enable the use of collaborative interactive machine learning for problem solving in a large scale.

show abstract

Section: Open Problemsmentioning

confidence: 99%

“…One important problem which we have to face in future research is which questions to pose to humans and how to ask those questions [40]. At this point, human machineinteraction could provide useful insights and offer guidelines for the design of interfaces and visualisations.…”

Section: Open Problemsmentioning

confidence: 99%

Reasoning Under Uncertainty: Towards Collaborative Interactive Machine Learning

Robert

Büttner

Röcker

et al. 2016

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…tools for efficient video annotation [53] and object labelling [34]. Methods that intelligently design the query space [39,32,30] also share the spirit of reducing annotation effort. Other works have looked into active learning schemes that query for multiple types of annotator feedback [50,4,43].…”

Section: Related Workmentioning

confidence: 99%

Active Learning for Human Pose Estimation

Liu

Ferrari

2017

2017 IEEE International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

Annotating human poses in realistic scenes is very time consuming, yet necessary for training human pose estimators. We propose to address this problem in an active learning framework, which alternates between requesting the most useful annotations among a large set of unlabelled images, and re-training the pose estimator. To this end, (1) we propose an uncertainty estimator specific for body joint predictions, which takes into account the spatial distribution of the responses of the current pose estimator on the unlabelled images; (2) we propose a dynamic combination of influence and uncertainty cues, where their weights vary during the active learning process according to the reliability of the current pose estimator; (3) we introduce a computer assisted annotation interface, which reduces the time necessary for a human annotator to click on a joint by discretizing the image into regions generated by the current pose estimator. Experiments using the MPII and LSP datasets with both simulated and real annotators show that (1) the proposed active selection scheme outperforms several baselines; (2) our computer-assisted interface can further reduce annotation effort; and (3) our technique can further improve the performance of a pose estimator even when starting from an already strong one.

show abstract

“…integrate computer vision and crowdsourcing is to understand the performance of computer vision algorithms and adaptively use crowd work [158,215]. In chapter 5, we described a method to use an ML-based supervised workflow controller to assess the difficulty of each object detection task, which allowed us to adaptively allocate work to different crowd workflows to reduce human cost.…”

Section: Combining Of Crowdsourcing and Computer Vision The Key To Ementioning

confidence: 99%

“…For example, Zhang et al used a supervised machine learning algorithms to detect computer vision failure in semantic image segmentation and vanishing point detection [215]. Russakovsky et al introduced a method to combine a variety of crowdsourcing tasks with object detection algorithms using Markov decision process that automatically balances human cost and object detection accuracy [158].…”

Section: Combining Of Crowdsourcing and Computer Vision The Key To Ementioning

confidence: 99%

Scalable methods to collect and visualize sidewalk accessibility data for people with mobility impairments

Hara

2014

Proceedings of the Adjunct Publication of the 27th Annual ACM Symposium on User Interface Software and Technology - UIST'14 Adj

View full text Add to dashboard Cite

Poorly maintained sidewalks pose considerable accessibility challenges for people with mobility impairments. Despite comprehensive civil rights legislation of Americans with Disabilities Act, many city streets and sidewalks in the U.S. remain inaccessible.The problem is not just that sidewalk accessibility fundamentally affects where and how people travel in cities, but also that there are few, if any, mechanisms to determine accessible areas of a city a priori.To address this problem, my Ph.D. dissertation introduces and evaluates new scalable methods for collecting data about street-level accessibility using a combination of crowdsourcing, automated methods, and Google Street View (GSV). My dissertation has four research threads. First, we conduct a formative interview study to establish a better understanding of how people with mobility impairments currently assess accessibility in the built environment and the role of emerging location-based technologies therein. The study uncovers the existing methods for assessing accessibility of physical environment and identify useful features of future assistive technologies. Second, we develop and evaluate scalable crowdsourced accessibility data collection methods. We show that paid crowd workers recruited from an online labor marketplace can find and label accessibility attributes in GSV with accuracy of 81%. This accuracy improves to 93% with quality control mechanisms such as majority vote. Third, we design a system that combines crowdsourcing and automated methods to increase data collection efficiency. Our work shows that by combining crowdsourcing and automated methods, we can increase data collection efficiency by 13% without sacrificing accuracy. Fourth, we develop and deploy a web tool that lets volunteers to help us collect the street-level accessibility data from Washington, D.C.As of writing this dissertation, we have collected the accessibility data from 20% of the streets in D.C. We conduct a preliminary evaluation on how the said web tool is used.Finally, we implement proof-of-concept accessibility-aware applications with accessibility data collected with the help of volunteers.My dissertation contributes to the accessibility, computer science, and HCI communities by: (i) extending the knowledge of how people with mobility impairments interact with technology to navigate in cities; (ii) introducing the first work that demonstrates that GSV is a viable source for learning about the accessibility of the physical world; (iii) introducing the first method that combines crowdsourcing and automated methods to remotely collect accessibility information; (iv) deploying interactive web tools that allow volunteers to help populate the largest dataset about street-level accessibility of the world; and (v) demonstrating accessibility-aware applications that empower people with mobility impairments.

show abstract

Best of both worlds: Human-machine collaboration for object annotation

Cited by 193 publications

References 54 publications

Reasoning Under Uncertainty: Towards Collaborative Interactive Machine Learning

Reasoning Under Uncertainty: Towards Collaborative Interactive Machine Learning

Active Learning for Human Pose Estimation

Scalable methods to collect and visualize sidewalk accessibility data for people with mobility impairments

Contact Info

Product

Resources

About