Unsupervised Keyphrase Extraction for Web Pages

Abstract

Keyphrase extraction is an important part of natural language processing (NLP) research, although little research is done in the domain of web pages. The World Wide Web contains billions of pages that are potentially interesting for various NLP tasks, yet it remains largely untouched in scientific research. Current research is often only applied to clean corpora such as abstracts and articles from academic journals or sets of scraped texts from a single domain. However, textual data from web pages differ from normal text documents, as it is structured using HTML elements and often consists of many small fragments. These elements are furthermore used in a highly inconsistent manner and are likely to contain noise. We evaluated the keyphrases extracted by several state-of-the-art extraction methods and found that they did not transfer well to web pages. We therefore propose WebEmbedRank, an adaptation of a recently proposed extraction method that can make use of structural information in web pages in a robust manner. We compared this novel method to other baselines and state-of-the-art methods using a manually annotated dataset and found that WebEmbedRank achieved significant improvements over existing extraction methods on web pages.

Phrase indexing and the identification of related academic research content

“…As an extension to the index navigation application, a further goal is to develop and implement 'see also' functionality, so that when a user selects a particular file, the application suggests and provides links to related files in the indexed corpus. This kind of functionality, now widespread and familiar, is one outcome of the half a century of fundamental research into information retrieval to which an extensive literature attests (e.g., in relation to the present investigations, refs [1][2][3][4][5][6][7][8][9]. Increasingly, such functionality is delivered by AI-based approaches such as those developed by, for example, UNSILO (https://unsilo.ai) and Yewno (https://www.yewno.com).…”

Phrase indexing and the identification of related academic research content

“…As an extension to the index navigation application, a further goal is to develop and implement 'see also' functionality, so that when a user selects a particular file, the application suggests and provides links to related files in the indexed corpus. This kind of functionality, which is now widespread and familiar, represents the fruition of half a century of fundamental research into information retrieval, to which an extensive literature attests (e.g., in relation to the present investigations, refs [1][2][3][4][5][6][7][8][9]. Increasingly, such functionality is delivered by AI-based technologies such as those developed by, for example, UNSILO (https://unsilo.ai) and Yewno (https://www.yewno.com).…”

“…def find_intersection(dictA, dictB): # Return the elements that are common to dictA and dictB # Inputs are now dictionaries of the form {phr_id: signif, ... } intersect = [] sum_A_sigs = 0.0 sum_B_sigs = 0.0 for key in dictA: # key is a phrase ID sig1 = dictA[key] # sig1: significance in file A of phrase with ID value = key if key in dictB: sig2 = dictB[key] # sig2: significance in file B of phrase with ID value = key intersect.append( [key] ) # add to intersection if phrase is common to A and B sum_A_sigs = sum_A_sigs + sig1 sum_B_sigs = sum_B_sigs + sig2 # Compute the significance-based weighting by which to multiply the A-B link strength # -the product of the average significance values for A and B: intersect_size = len(intersect) av_A_sig = sum_A_sigs / (1+intersect_size) av_B_sig = sum_B_sigs / (1+intersect_size) tot_sig = av_A_sig * av_B_sig # Return the intersection size and combined significance: return (intersect_size, tot_sig) (vi) The overall similarity between files A and B is calculated as the product of an adjusted version of the total significance, taking into account phrase frequency statistics and the number of phrase components as outlined above, and the Overlap Multiplication (OM) term already discussed: 4 AB_score = (math.sqrt(tot_sig) * 100) * ( intersect**2 / ((1+A_only) * (1+B_only)) )…”

“…Initially standard Python lists were used for storing each file's phrase ID/phrase significance pairs, and to find the intersection involved nested for loops to iterate over the files A (outer loop) and then, for each file A, over the files B (inner loop). Switching to the dictionary-based approach outlined above reduced the time taken to compute the similarities for the test corpus by very roughly an order of magnitude 4. The adjustment to the significance (involving square root and multiplication operations) was introduced pragmatically in order to temper the impact of the significance weighting on the overall measure, when initial experiments raised the possibility that its effect in combination with the OM term was perhaps excessive.…”

Keyphrases Concentrated Area Identification from Academic Articles as Feature of Keyphrase Extraction: A New Unsupervised Approach

“…Various keyphrase extraction methods have been developed to support the aforementioned applications [8], [9], [7], [10], [11], [12]. Domain-specific strategies [9], for example, need knowledge of the application domain, whereas linguistic approaches [9] demand language proficiency.…”