An automation algorithm for harvesting capital market information from the web

Abstract

Purpose-The purpose of this paper is to develop an algorithm to harvest user specified information on finance portals and compile it into machine-readable datasets for quantitative analysis. Design/methodology/approach-The Visual Basic macro language in Microsoft Excel is applied to develop code that is not constrained by the single-query function of Excel. The core of the algorithm is built around the splitting of the URL connector line and the placement of a continuously updating variable into which are looped as many tickers as there are in the input list. The output is then written to non-overlapping cells. Findings-Numerical information placed on major finance websites can be harvested into structured machine-readable datasets by applying this algorithm. Research limitations/implications-One significant change in Microsoft Excel 2007 is that the worksheet is expanded from 2 24 to 2 34 cells, or to be more specific, from 256 (IV) columns  65,536 rows (2 8  2 16) to 16,384 (XFD)  1,048,576 (2 14  2 20). These new limits while allowing for a larger number of tickers, still constrain a single worksheet to 16,384 columns. For five fields per ticker that translates into roughly 3,200 ticker symbols. Practical implications-The algorithm extends user accessibility to websites that do not provide the facility of simultaneous downloading of information on multiple stock tickers. Furthermore, the procedure automates the downloading of multiple pieces of information (fields) and entire tables per ticker (record). Originality/value-An exhaustive literature search did not find any paper that discusses a multiple ticker algorithm for web harvesting.

A bibliometric analysis of digital financial literacy

“…Even those who do not know how to code can utilise it. Similarly, Agrrawal (2009), Managerial Finance, developed and applied an automation algorithm to harvest machine-readable financial datasets.…”

Identification of the Dynamic Trade Relationship between China and the United States Using the Quantile Grey Lotka–Volterra Model

“…Under the minimum MAPE principle, the non-linear grey Bernoulli model parameters were estimated in the particle swarm optimised (PSO) algorithm. Automation algorithms were deployed by Agrrawal [34] to derive capital market information. Wang and Hsu [35] estimated the parameters of the GM(1,1) model used with genetic algorithms.…”

The Gibbons, Ross, and Shanken Test for Portfolio Efficiency: A Note Based on Its Trigonometric Properties

“…the GRS test is fundamental for testing MV (mean-variance) efficiency under normality." He uses the statistic and rejects the MV efficiency on the CRSP (Center for Research in Security Prices, WRDS [26]; Agrrawal [27]) value-weighted stock index for three of the six consecutive ten-year sub-periods from 1926 to 1986 besides rejecting the normality assumption of the data at the same time. Zhou [25] conducts the zero-intercept test on the CAPM under the normality as well as the alternative (elliptical) distribution assumption and uses the values of the GRS statistic to calculate the p-values (level of significance).…”