Evaluating how developers use general-purpose web-search for code retrieval

Abstract

Search is an integral part of a software development process. Developers often use search engines to look for information during development, including reusable code snippets, API understanding, and reference examples. Developers tend to prefer generalpurpose search engines like Google, which are often not optimized for code related documents and use search strategies and ranking techniques that are more optimized for generic, non-code related information.In this paper, we explore whether a general purpose search engine like Google is an optimal choice for code-related searches. In particular, we investigate whether the performance of searching with Google varies for code vs. non-code related searches. To analyze this, we collect search logs from 310 developers that contains nearly 150,000 search queries from Google and the associated result clicks. To differentiate between code-related searches and non-code-related searches, we build a model which identifies the code intent of queries. Leveraging this model, we build an automatic classifier that detects a code and non-code related query. We confirm the effectiveness of the classifier on manually annotated queries where the classifier achieves a precision of 87%, a recall of 86%, and an F1-score of 87%. We apply this classifier to automatically annotate all the queries in the dataset. Analyzing this dataset, we observe that code related searching often requires more effort (e.g., time, result clicks, and query modifications) than general noncode search, which indicates code search performance with a general search engine is less effective. ACM Reference format:

Exploring Web Search Engines to Find Architectural Knowledge

“…searching for solutions to bugs or for third party libraries) using web search engines. Rahman et al [29] studied the effectiveness of web searching when searching for source code on the web. Their results show that searching for source code is harder than searching for other types.…”

Guided pattern mining for API misuse detection by change-based code analysis

“…This indicates that the code search is heavily domain and use-case-specific. However, still, developers tend to prefer general-purpose search engines for code search (Sadowski et al 2015;Rahman et al 2018). In contrast, our goal was to find similar code examples without requiring human interaction.…”

Search4Code: Code Search Intent Classification Using Weak Supervision

“…Wang et al have leveraged intent understanding for improving effort estimation in code reviews [17], [18]. Recently, software engineering related search queries have been analyzed and classified into different categories by using distant supervision [7] and tokenlevel intent aggregation [8]. Our goal is to further improve upon these methods by introducing a weak supervision based approach for code search intent classification.…”

“…In order to create a large-scale dataset of code search queries, it is crucial to automatically detect code search intent in search queries. Previous research in the area of search query classification [7], [8] has focused primarily on classification of web queries in categories such as Debug, API, and HowTo using heuristics and rule-based methods which tend to overfit.…”