Improving Long Distance Slot Carryover in Spoken Dialogue Systems

Abstract

Tracking the state of the conversation is a central component in task-oriented spoken dialogue systems. One such approach for tracking the dialogue state is slot carryover, where a model makes a binary decision if a slot from the context is relevant to the current turn. Previous work on the slot carryover task used models that made independent decisions for each slot. A close analysis of the results show that this approach results in poor performance over longer context dialogues. In this paper, we propose to jointly model the slots. We propose two neural network architectures, one based on pointer networks that incorporate slot ordering information, and the other based on transformer networks that uses self attention mechanism to model the slot interdependencies. Our experiments on an internal dialogue benchmark dataset and on the public DSTC2 dataset demonstrate that our proposed models are able to resolve longer distance slot references and are able to achieve competitive performance. x 1 x 2 x 3 x 4 I i c ✓ ✗ ✗ ✓ Decoder Intent Encoder Dialogue Encoder (bi-LSTM) Serialized Dialogue Token Embeddings Contextual Embeddings h A 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " h q y x w m I P y D k y 4 H n J D 7 F b g 9 t Z G Q o = " > A A A C A n i c b V D L T g I x F L 2 D L 8 Q X 6 t J N I z F x R W a M i S 5 R N y 4 x c Y A E R t I p H W h o O 5 O 2 Y y S E n X u 3 + g v u j F t / x D / w M + z A L A Q 8 S Z O T c + 6 r J 0 w 4 0 8 Z 1 v 5 3 C y u r a + k Z x s 7 S 1 v b O 7 V 9 4 / a O g 4 V Y T 6 J O a x a o V Y U 8 4 k 9 Q 0 z n L Y S R b E I O W 2 G w 5 v M b z 5 S p V k s 7 8 0 o o Y H A f c k i R r C x k j / o e g 9 X 3 X L F r b p T o G X i 5 a Q C O e r d 8 k + n F 5 N U U G k I x 1 q 3 P T c x w R g r w w i n k 1 I n 1 T T B Z I j 7 t G 2 p x I L q Y D w 9 d o J O r N J D U a z s k w Z N 1 b 8 d Y y y 0 H o a 1 _ b a s e 6 4 = " h q y x w m I P y D k y 4 H n Je g 9 X 3 X L F r b p T o G X i 5 a Q C O e r d 8 k + n F 5 N U U G k I x 1 q 3 P T c x w R g r w w i n k 1 I n 1 T T B Z I j 7 t G 2 p x I L q Y D w 9 d o J O r N J D U a z s k w Z N 1 b 8 d Y y y 0 H o a 1 _ b a s e 6 4 = " h q y x w m I P y D k y 4 H n J D 7 F b g 9 t Z G Q o = " > A A A C A n i c b V D L T g I x F L 2 D L 8 Q X 6 t J N I z F x R W a M i S 5 R N y 4 x c Y A E R t I p H W h o O 5 O 2 Y y S E n X u 3 + g v u j F t / x D / w M + z A L A Q 8 S Z O T c + 6 r J 0 w 4 0 8 Z 1 v 5 3 C y u r a + k Z x s 7 S 1 v b O 7 V 9 4 / a O g 4 V Y T 6 J O a x a o V Y U 8 4 k 9 Q 0 z n L Y S R b E I O W 2 G w 5 v M b z 5 S p V k s 7 8 0 o o Y H A f c k i R r C x k j / o e g 9 X 3 X L F r b p T o G X i 5 a Q C O e r d 8 k + n F 5 N U U G k I x 1 q 3 P T c x w R g r w w i n k 1 I n 1 T T B Z I j 7 t G 2 p x I L q Y D w 9 d o J O r N J D U a z s k w Z N 1 b 8 d Y y y 0 H o a 1 _ b a s e 6 4 = " 3 K N N 2 9 b r 2 o 9 T 8 I 1 E R 0 O N a m 5 p 2 0 I = " > A A A C A n i c b V D L S g M x F L 1 T X 3 W s W t d u g k V w V W b c 6 F J w 4 7 K C f U B b S i a T a U O T z J D c E c v Q H 3 A n f o w 7 8 T f 8 F j d m 2 i 5 s 6 4 U L h 3 M S z r k n y q S w G A T...

Joint Modeling of Arguments for Event Understanding

“…Intent type Event type We propose a novel model for joint modeling of potential arguments inspired by Chen et al (2019) for slot-filling in dialogue systems, which proposed to jointly predict spans that are relevant to the intent of the current round of dialogue. Over detected arguments, a Transformer (Vaswani et al, 2017) encoder is placed upon the event trigger and potential arguments to jointly learn the relations between the event trigger and its arguments.…”

“…Dhingra et al (2017) augmented such methods with external knowledge bases (KBs) to create a multi-turn dialogue agent which helps users search KBs. Chen et al (2019) proposed joint models over potential slots in dialogue to output which contextual slots should be carried over to the most recent utterance. Our approach is inspired by this work, by drawing analogies between concepts in SLU (intents / slots) and those in IE (events / arguments) (see Table 1).…”

“…A Transformer, by its self-attention mechanism, naturally models the relation between every trigger-argument and argument-argument pair. Note two major differences as compared to a Transformer that runs on tokens: (1) each input to the Transformer represents a span instead of a token, following Chen et al (2019); (2) since the arguments do not take an explicit sequential order, we forgo the positional embeddings in Transformers, effectively modeling the input as a set of spans instead of a sequence (self-attention exhibits the property of permutation invariance without positional embeddings ).…”

“…E.g., fromChen et al (2019): What's the weather in San Francisco? ... Any good Mexican restaurants there?3 https://nlp.jhu.edu/rams.…”

“…According toChen et al (2019), increasing the number of attention heads substantially improves the model performance, so we prefer more attention heads over more encoder layers.…”