Augmenting OAI-PMH repository holdings using search engine APIs

Klein, Martin; Nelson, Michael L.; Pao, Juliet Z.

doi:10.1145/1255175.1255285

Cited by 1 publication

(1 citation statement)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While many DLs still use this technique, focused crawling is now often augmented [25,42,52], or even replaced [27,30], with search engine APIs. Depending on their collection development policy, DL administrators should be informed about the issues of completeness and synchronization between the API and WUI interfaces.…”

Section: Research Use Of the Apismentioning

confidence: 99%

Agreeing to disagree

McCown

Nelson

2007

Proceedings of the 7th ACM/IEEE-CS Joint Conference on Digital Libraries

Self Cite

View full text Add to dashboard Cite

Google, Yahoo and MSN all provide both web user interfaces (WUIs) and application programming interfaces (APIs) to their collections. Whether building collections of resources or studying the search engines themselves, the search engines request that researchers use their APIs and not "scrape" the WUIs. However, anecdotal evidence suggests the interfaces produce different results. We provide the first in depth quantitative analysis of the results produced by the Google, MSN and Yahoo API and WUI interfaces. We have queried both interfaces for five months and found significant discrepancies between the interfaces in several categories. In general, we found MSN to produce the most consistent results between their two interfaces. Our findings suggest that the API indexes are not older, but they are probably smaller for Google and Yahoo. We also examined how search results decay over time and built predictive models based on the observed decay rates. Based on our findings, it can take over a year for half of the top 10 results to a popular query to be replaced in Google and Yahoo; for MSN it may take only 2-3 months.

show abstract

Section: Research Use Of the Apismentioning

confidence: 99%