Design and evaluation of a multi-agent collaborative Web mining system

Chau, Michael; Zeng, Daniel; Chen, Hsinchun; Huang, Michael Bailou; Hendriawan, David

doi:10.1016/s0167-9236(02)00103-3

Cited by 79 publications

(32 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A web crawler, known as a network robot or spider, is a program, software, or programmed script that automatically fetches large collections of web pages according to http protocol and some clearly defined crawling strategies (e.g., depth-first strategy, breadth-first strategy, and best-first strategy) [22,23]. It can be categorized into general purpose web crawlers and focused web crawlers according to the purposes of the tools.…”

Section: Web Crawler Technologymentioning

confidence: 99%

Using Web Crawler Technology for Geo-Events Analysis: A Case Study of the Huangyan Island Incident

Hou

2014

Sustainability

View full text Add to dashboard Cite

Social networking and network socialization provide abundant text information and social relationships into our daily lives. Making full use of these data in the big data era is of great significance for us to better understand the changing world and the information-based society. Though politics have been integrally involved in the hyperlinked world issues since the 1990s, the text analysis and data visualization of geo-events faced the bottleneck of traditional manual analysis. Though automatic assembly of different geospatial web and distributed geospatial information systems utilizing service chaining have been explored and built recently, the data mining and information collection are not comprehensive enough because of the sensibility, complexity, relativity, timeliness, and unexpected characteristics of political events. Based on the framework of Heritrix and the analysis of web-based text, word frequency, sentiment tendency, and dissemination path of the Huangyan Island incident were studied by using web crawler technology and the text analysis. The results indicate that tag cloud, frequency map, attitudes pie, individual mention ratios, and dissemination flow graph, based on the crawled information and data processing not only highlight the characteristics of geo-event itself, but also implicate many interesting phenomenon and deep-seated problems behind it, such as related topics, theme vocabularies, subject contents, hot countries, event bodies, opinion leaders, high-frequency vocabularies, information sources, semantic structure, propagation paths, distribution of different attitudes, and regional difference of net citizens' response in the Huangyan Island incident. Furthermore, the text analysis of network information with the help of focused web crawler OPEN ACCESSSustainability 2014, 6 1897 is able to express the time-space relationship of crawled information and the information characteristic of semantic network to the geo-events. Therefore, it is a useful tool to collect information for understanding the formation and diffusion of web-based public opinions in political events.

show abstract

Section: Web Crawler Technologymentioning

confidence: 99%

Using Web Crawler Technology for Geo-Events Analysis: A Case Study of the Huangyan Island Incident

Hou

2014

Sustainability

View full text Add to dashboard Cite

show abstract

“…A web crawler, also known as a network robot or spider, is a kind of script or programme that can automatically traverse the internet and grab text information based on the web link structure of World Wide Web and HTTP protocols (Chau et al 2003). Web crawler search engines originated in 1990s and have become a hot topic in today's search engine and web mining fields (Liu et al 2007).…”

Section: Web Crawler Technologymentioning

confidence: 99%

Using Web Crawler Technology for Text Analysis of Geo-Events: A Case Study of the Huangyan Island Incident

2013

Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci.

View full text Add to dashboard Cite

Commission KEY WORDS: web crawler technology; text information; sentiment analysis; Huangyan Island incident ABSTRACT:With the social networking and network socialisation have brought more text information and social relationships into our daily lives, the question of whether big data can be fully used to study the phenomenon and discipline of natural sciences has prompted many specialists and scholars to innovate their research. Though politics were integrally involved in the hyperlinked word issues since 1990s, automatic assembly of different geospatial web and distributed geospatial information systems utilizing service chaining have explored and built recently, the information collection and data visualisation of geo-events have always faced the bottleneck of traditional manual analysis because of the sensibility, complexity, relativity, timeliness and unexpected characteristics of political events. Based on the framework of Heritrix and the analysis of web-based text, word frequency, sentiment tendency and dissemination path of the Huangyan Island incident is studied here by combining web crawler technology and the text analysis method. The results indicate that tag cloud, frequency map, attitudes pie, individual mention ratios and dissemination flow graph based on the data collection and processing not only highlight the subject and theme vocabularies of related topics but also certain issues and problems behind it. Being able to express the time-space relationship of text information and to disseminate the information regarding geo-events, the text analysis of network information based on focused web crawler technology can be a tool for understanding the formation and diffusion of web-based public opinions in political events.

show abstract

“…Web content mining refers to the discovery of useful information from Web contents, including text, images, audio, video, and so on. Web content mining research includes resource discovery from the Web (e.g., Chakrabarti, van den Berg, & Dom, 1999;Chau, Zeng, Chen, Huang, & Hendriawan, 2003), document categorization and clustering (e.g., Chen, Fan, Chau, & Zeng, 2001;Kohonen et al, 2000;Zamir & Etzioni, 1999), and information extraction from Web pages (e.g., Hurst, 2001). Web structure mining studies the model underlying the link structures of the Web.…”

Section: Related Workmentioning

confidence: 99%

Web searching in Chinese: A study of a search engine in Hong Kong

Chau¹,

Fang²,

Yang³

2007

J. Am. Soc. Inf. Sci.

Self Cite

View full text Add to dashboard Cite

The number of non-English resources has been increasing rapidly on the Web. Although many studies have been conducted on the query logs in search engines that are primarily English-based (e.g., Excite and AltaVista), only a few of them have studied the information-seeking behavior on the Web in non-English languages. In this article, we report the analysis of the search-query logs of a search engine that focused on Chinese. Three months of search-query logs of Timway, a search engine based in Hong Kong, were collected and analyzed. Metrics on sessions, queries, search topics, and character usage are reported. N-gram analysis also has been applied to perform character-based analysis. Our analysis suggests that some characteristics identified in the search log, such as search topics and the mean number of queries per sessions, are similar to those in English search engines; however, other characteristics, such as the use of operators in query formulation, are significantly different. The analysis also shows that only a very small number of unique Chinese characters are used in search queries. We believe the findings from this study have provided some insights into further research in non-English Web searching.

show abstract

Design and evaluation of a multi-agent collaborative Web mining system

Cited by 79 publications

References 44 publications

Using Web Crawler Technology for Geo-Events Analysis: A Case Study of the Huangyan Island Incident

Using Web Crawler Technology for Geo-Events Analysis: A Case Study of the Huangyan Island Incident

Using Web Crawler Technology for Text Analysis of Geo-Events: A Case Study of the Huangyan Island Incident

Web searching in Chinese: A study of a search engine in Hong Kong

Contact Info

Product

Resources

About