Large-Context Conversational Representation Learning: Self-Supervised Learning For Conversational Documents

Hierarchical Knowledge Distillation for Dialogue Sequence Labeling

“…Utterance-level dialogue sequence labeling is being used for topic segmentation [7][8][9], dialogue act estimation [10][11][12][13][14][15], and call scene segmentation [16][17][18]. Hierarchically structured models consisting of utterance-level and dialogue-level neural networks are often used to efficiently capture contexts within an utterance and between utterances, and an effective self-supervised pretraining method has been proposed [18]. If a hierarchical model is used for dialogue sequence labeling, a large number of parameters are needed to train a model that offers high accuracy.…”

“…When self-supervised pretraining [18] is utilized, parameters {θ w , θ r , θ s , θ u } are initialized by pretraining using unlabeled data, and then parameters Θ are optimized with L HT in the same way as above.…”

“…In this paper, we focus on utterance-level dialogue sequence labeling, a key component in dialogue document understanding. Dialogue sequence labeling is often modeled as a supervised learning task that estimates labels for each utterance when given a dialogue document; it is useful in many applications such as topic segmentation [7][8][9], dialogue act estimation [10][11][12][13][14][15], and call scene segmentation [16][17][18]. To understand dialogue documents, it is necessary to consider who spoke what and in what order.…”

“…Therefore, these techniques often adopt a hierarchically-structured model consisting of an utterance-level network and a dialogue-level network to capture contexts not only within an utterance but also between utterances [16]. In addition, an effective selfsupervised pretraining method using only unlabeled data has been proposed [18].…”

“…• We conduct ablation experiments on dialogue act estimation and call scene segmentation tasks that analyze the effectiveness of the proposed method. We also provide the results achieved by combining self-supervised pretraining [18] and the proposed method.…”