Text characteristics of English language university Web sites

Abstract

The nature of the contents of academic Web sites is of direct relevance to the new field of scientific Web intelligence, and for search engine and topic-specific crawler designers. We analyze word frequencies in national academic Webs using the Web sites of three Englishspeaking nations: Australia, New Zealand, and the United Kingdom. Strong regularities were found in page size and word frequency distributions, but with significant anomalies. At least 26% of pages contain no words. High frequency words include university names and acronyms, Internet terminology, and computing product names: not always words in common usage away from the Web. A minority of low frequency words are spelling mistakes, with other common types including nonwords, proper names, foreign language terms or computer science variable names. Based upon these findings, recommendations for data cleansing and filtering are made, particularly for clustering applications.

MySpace comments

“…Although the difference is small in size, it is large due to the logarithmic scale. This is in contrast to similar graphs for British English and academic websites, for example, in which the lines are straight and the point for word frequency 1 does not deviate (Thelwall, 2005). This confirms that there are more unique words in comments than in “normal” text.…”

Language evolution and the spread of ideas on the Web: A procedure for identifying emergent hybrid word family members

“…The straight line graph in Figure 2 reflects a power law that is common in word frequency statistics (Li, 1992; Zipf, 1949) and found almost everywhere on the Web (Baldi et al, 2003; Barabási, 2002; Thelwall, 2005), but the nonlinear Figure 1 is more surprising, showing that there is an unnaturally high number of low‐frequency GM words. This would be consistent with a set of activists' attempting to promote the usage of certain words, but these words' not gaining a resonance with the public.…”

“…Statistical analyses of Web patterns for many Web‐related phenomena, including page sizes, link counts, and word usage patterns, often show small numbers of documents dominating, an aspect of the “power law” effect (Baldi, Frasconi, & Smyth, 2003; Barabási, 2002; Levene & Poulovassilis, 2004; Thelwall, 2005; Zipf, 1949). This suggests that an effective strategy for finding hybrid word family members would be to look for individual pages that contain many such terms, which may be seen as authorities on the topic and hence likely to contain many different related words.…”

Web issue analysis: An integrated water resource management case study

“…A few Web text analysis studies have investigated the text of Web sites from an issue perspective, by using either content analysis (Weare & Lin, 2000) or metrics based on word frequency counting (Price & Thelwall, 2005; Thelwall, 2004c, 2005). Although content analyses can yield detailed and insightful information, they are labor‐intensive and time‐consuming.…”