Web mining for innovation ecosystem mapping: a framework and a large-scale pilot study

Abstract

Existing approaches to model innovation ecosystems have been mostly restricted to qualitative and small-scale levels or, when relying on traditional innovation indicators such as patents and questionnaire-based survey, suffered from a lack of timeliness, granularity, and coverage. Websites of firms are a particularly interesting data source for innovation research, as they are used for publishing information about potentially innovative products, services, and cooperation with other firms. Analyzing the textual and relational content on these websites and extracting innovation-related information from them has the potential to provide researchers and policy-makers with a cost-effective way to survey millions of businesses and gain insights into their innovation activity, their cooperation, and applied technologies. For this purpose, we propose a web mining framework for consistent and reproducible mapping of innovation ecosystems. In a large-scale pilot study we use a database with 2.4 million German firms to test our framework and explore firm websites as a data source. Thereby we put particular emphasis on the investigation of a potential bias when surveying innovation systems through firm websites if only certain firm types can be surveyed using our proposed approach. We find that the availability of a websites and the characteristics of the website (number of subpages and hyperlinks, text volume, language used) differs according to firm size, age, location, and sector. We also find that patenting firms will be overrepresented in web mining studies. Web mining as a survey method also has to cope with extremely large and hyper-connected outlier websites and the fact that low broadband availability appears to prevent some firms from operating their own website and thus excludes them from web mining analysis. We then apply the proposed framework to map an exemplary innovation ecosystem of Berlin-based firms that are engaged in artificial intelligence. Finally, we outline several approaches how to transfer firm website content into valuable innovation indicators.

An Integrated Data Framework for Policy Guidance in Times of Dynamic Economic Shocks

“…First and foremost, not all companies have their own corporate website domain, which likely biases our web-based analysis results. Previous studies have shown that URL coverage of German companies is at 46% (Kinne & Axenbeck 2020). Especially among smaller firms the fraction without corporate URL is comparatively high.…”

“…The extraction of relevant information from unstructured or semi-structured text data from corporate websites can be seen as particularly challenging here. At the same time, it promises a number of benefits, particularly in terms of granularity, timeliness, scope and cost of collection (Kinne & Axenbeck 2020). These benefits will also turn out to be key in this study.…”

“…For each corporate website address a maximum of five webpages per company (a website usually consists of several webpages) were crawled. The selection of these webpages was not conducted at random, but followed a clear heuristic: first, webpages with the shortest Uniform Resource Locator (URL) within the corporate website domain and whose content is written in German were selected (see Kinne and Axenbeck (2020) for more details on the scraping framework). The former selection criteria satisfies that those webpages with more general and up-to-date ('top-level') information were downloaded with priority making it more likely that recent Corona references are captured by the search query.…”

Ecosystem Intelligence for AI-based Assistant Platforms

“…However, neither are details on data collection nor on processing provided, and only a few approaches use web mining to scrape unstructured information from enterprise websites (e.g. [28]).…”

“…Instead, to collect data for ecosystem intelligence, a wide range of approaches are used, starting with analyzing enterprise websites to analyze the logo maps [29]. In [28], a web-mining approach is used to collect ecosystem data from enterprises. In [24], the authors combine a variety of independently created data sources on deals and alliance relationships, executive and funding relationships (EFRs), and public opinion and discourse (POD) to perform the analysis of ecosystems.…”