A Forecasting Model for the Pages Crawled by Search Engine Crawlers at a Web Site

Abstract

World Wide Web is exploding in terms of the number of web sites and users. Without search engines the web sites will not be visible to the users. Different search engine crawlers behave in different ways while they access a web site. The number of visits and pages crawled by search engines could be helpful in identifying their behavior and also the server load. A forecasting model in time series has been proposed for predicting the number of pages crawled by search engines. This model was compared with the actual values and it was found feasible. General TermsWeb log mining, Web analytics

Application of ARIMA(1,1,0) Model for Predicting Time Delay of Search Engine Crawlers

“…There are several works that mentions about the search engine crawler behavior. A forecasting model is proposed for the number of pages crawled by search engine crawlers at a web site [3]. Sun et al has conducted a large scale study of robots.txt [2].…”

“…There is open source software available like Google Analytics which measures the number of visitors, duration of the visits, the demographic from which the visitor comes etc. But it cannot identify search engine visits because Google Analytics track users with the help of JavaScript and search engine crawlers do not enable the JavaScript embedded in web pages when the crawlers visit the web sites [3]. The search engine crawlers initially access the robots.txt file which specifies the Robot Exclusion Protocol.…”