Automatic story segmentation using a Bayesian decision framework for statistical models of lexical chain features

Abstract

This paper presents a Bayesian decision framework that performs automatic story segmentation based on statistical modeling of one or more lexical chain features. Automatic story segmentation aims to locate the instances in time where a story ends and another begins. A lexical chain is formed by linking coherent lexical items chronologically. A story boundary is often associated with a significant number of lexical chains ending before it, starting after it, as well as a low count of chains continuing through it. We devise a Bayesian framework to capture such behavior, using the lexical chain features of start, continuation and end. In the scoring criteria, lexical chain starts/ends are modeled statistically with the Weibull and uniform distributions at story boundaries and non-boundaries respectively. The normal distribution is used for lexical chain continuations. Full combination of all lexical chain features gave the best performance (F1=0.6356). We found that modeling chain continuations contributes significantly towards segmentation performance.

Laplacian Eigenmaps for Automatic Story Segmentation of Broadcast News

“…Substantial work has been reported in the story segmentation literature. In general, approaches can be categorized to detection-based [6], [9]- [11], [13], [17], [18], and model-based [5], [8], [19], [20]. Detection-based methods directly locate story 1558-7916/$31.00 © 2011 IEEE boundaries through a set of intuitive cues or features; a detector or classifier is learned from the cues to make boundary identification.…”

“…2, in real-world documents, we do not have a completely ideal situation where the connective strength between sentences in different stories is exactly 0, but if the connective strength between these sentences tend to be small, the Laplacian matrix can be treated as a perturbed version of the one in the ideal case. Formally, (13) where is the perturbation added to .…”

“…With the overwhelming growth of multimedia repositories from broadcast media and the Internet, researchers have been focusing on segmenting audio/video contents, such as broadcast news [6], meetings [7], and lectures [8], etc. Lexical cohesion-based methods have drawn much interest for story segmentation on texts and speech recognition transcripts of spoken documents [7], [9]- [13]. Lexical cohesion [14] indicates that words in a story (or topic) hang together by semantic relations and different stories tend to employ different sets of words.…”

Multi-modal feature integration for story boundary detection in broadcast news

“…Story boundary detection approaches can be categorized to detection-based [2]- [7] and model-based [8]- [10]. The former directly detects story boundaries through intuitive cues/features.…”

“…Among these previous studies, decision tree (DT) and maximum entropy (ME) model are frequently used as the boundary classification scheme. Recently, support vector machines (SVM) [5] and naive bayesian (NB) classifier [7] were adopted for story boundary detection. However, despite these tremendous efforts, we notice that a comprehensive comparison on different classifiers for multi-modal feature integration is still missing and the potential state-of-the-art story boundary detection performances remain unknown.…”