A deep joint sparse non-negative matrix factorization framework for identifying the common and subject-specific functional units of tongue motion during speech

Woo, Jonghye; Xing, Fangxu; Prince, Jerry L.; Stone, Maureen; Gomez, Arnold D.; Reese, Timothy G.; Wedeen, Van J.; Fakhri, Georges El

doi:10.1016/j.media.2021.102131

Cited by 11 publications

(7 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We then computed a sequence of voxel-level motion fields during the speech tasks from tagged MRI. 1,14 The data for this work were collected using a Siemens 3.0T TIM Trio system equipped with a 12-channel head coil and a 4-channel neck coil, using a segmented gradient echo sequence. 15,16 The 4D motion fields x has the size of 3 × 128 × 128 × 128 × 26.…”

Section: Resultsmentioning

confidence: 99%

“…This can help us associate tongue and oropharyngeal muscle deformation with its corresponding acoustic information. Internal tissue point tracking data from three-dimensional (3D) tagged MRI 1 sequences contain far more information about the tongue and oropharyngeal motion than does the more conventional two-dimensional (2D) mid-sagittal image sequences obtained from cine-MRI 2 and tagged MRI. 3 Yet, associating these four-dimensional (4D) deformation fields with speech audio waveforms poses the following challenges: 1) efficient feature extraction from complex and high-dimensional tongue and oropharyngeal deformation and 2) heterogeneous data representations between 4D motion fields and high-frequency one-dimensional (1D) audio waveforms.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Synthesizing audio from tongue motion during speech using tagged MRI via transformer

Liu¹,

Xing²,

Prince³

et al. 2023

Preprint

View full text Add to dashboard Cite

Investigating the relationship between internal tissue point motion of the tongue and oropharyngeal muscle deformation measured from tagged MRI and intelligible speech can aid in advancing speech motor control theories and developing novel treatment methods for speech related-disorders. However, elucidating the relationship between these two sources of information is challenging, due in part to the disparity in data structure between spatiotemporal motion fields (i.e., 4D motion fields) and one-dimensional audio waveforms. In this work, we present an efficient encoder-decoder translation network for exploring the predictive information inherent in 4D motion fields via 2D spectrograms as a surrogate of the audio data. Specifically, our encoder is based on 3D convolutional spatial modeling and transformer-based temporal modeling. The extracted features are processed by an asymmetric 2D convolution decoder to generate spectrograms that correspond to 4D motion fields. Furthermore, we incorporate a generative adversarial training approach into our framework to further improve synthesis quality on our generated spectrograms. We experiment on 63 paired motion field sequences and speech waveforms, demonstrating that our framework enables the generation of clear audio waveforms from a sequence of motion fields. Thus, our framework has the potential to improve our understanding of the relationship between these two modalities and inform the development of treatments for speech disorders.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Synthesizing audio from tongue motion during speech using tagged MRI via transformer

Liu¹,

Xing²,

Prince³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…We generated a sequence of voxel-level motion fields during the speech tasks from tagged MRI. 13,14 The resulting 4D motion fields x consist of 26 frames with a size of 3 × [(128 × 128 × 128) × 26], where each voxel of the motion fields has three channels to represent 3D directions. As for the corresponding audio waveforms, their lengths range from 21,832 to 24,175.…”

Section: Methodsmentioning

confidence: 99%

Speech motion anomaly detection via cross-modal translation of 4D motion fields from tagged MRI

Liu,

Xing,

Zhuo

et al. 2024

Medical Imaging 2024: Image Processing

View full text Add to dashboard Cite

Understanding the relationship between tongue motion patterns during speech and their resulting speech acoustic outcomes-i.e., articulatory-acoustic relation-is of great importance in assessing speech quality and developing innovative treatment and rehabilitative strategies. This is especially important when evaluating and detecting abnormal articulatory features in patients with speech-related disorders. In this work, we aim to develop a framework for detecting speech motion anomalies in conjunction with their corresponding speech acoustics. This is achieved through the use of a deep cross-modal translator trained on data from healthy individuals only, which bridges the gap between 4D motion fields obtained from tagged MRI and 2D spectrograms derived from speech acoustic data. The trained translator is used as an anomaly detector, by measuring the spectrogram reconstruction quality on healthy individuals or patients. In particular, the cross-modal translator is likely to yield limited generalization capabilities on patient data, which includes unseen out-of-distribution patterns and demonstrates subpar performance, when compared with healthy individuals. A one-class SVM is then used to distinguish the spectrograms of healthy individuals from those of patients. To validate our framework, we collected a total of 39 paired tagged MRI and speech waveforms, consisting of data from 36 healthy individuals and 3 tongue cancer patients. We used both 3D convolutional and transformer-based deep translation models, training them on the healthy training set and then applying them to both the healthy and patient testing sets. Our framework demonstrates a capability to detect abnormal patient data, thereby illustrating its potential in enhancing the understanding of the articulatory-acoustic relation for both healthy individuals and patients.

show abstract

“…The definition of deep unrolling was proposed by Gregor and LeCun [69], who unrolled the iterative shrinkage/thresholding algorithm (ISTA) to solve the optimization problem for sparse coding and achieved a nearly 20-fold improvement in time efficiency. Recently, by providing the neural network interpretability of iterative sparse coding with fewer layers and faster convergence, the ISTA-based deep unrolling algorithm has achieved great success in solv-ing inverse problems for biomedical imaging [70], exploiting multimodal side information for image superresolution [71], and implementing nonnegative matrix factorization for functional unit identification [72].…”

Section: Interpretable Deep Algorithm Unrollingmentioning

confidence: 99%

Working memory inspired hierarchical video decomposition with transformative representations

Qin¹,

Mao²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Video decomposition is very important to extract moving foreground objects from complex backgrounds in computer vision, machine learning, and medical imaging, e.g., extracting moving contrast-filled vessels from the complex and noisy backgrounds of X-ray coronary angiography (XCA). However, the challenges caused by dynamic backgrounds, overlapping heterogeneous environments and complex noises still exist in video decomposition. To solve these challenges, this study is the first to introduce a flexible visual working memory model in video decomposition to provide interpretable and high-performance hierarchical deep learning architecture, integrating the transformative representations between sensory and control layers from the perspective of visual and cognitive neuroscience. Specifically, robust PCA unrolling networks acting as a structure-regularized sensor layer decompose XCA into sparse/low-rank structured representations to separate moving contrast-filled vessels from noisy and complex backgrounds. Then, patch recurrent convolutional LSTM networks with a backprojection superresolution module embody unstructured random representations of the control layer in working memory, recurrently projecting spatiotemporally decomposed nonlocal patches into orthogonal subspaces for heterogeneous vessel retrieval and interference suppression. This video decomposition architecture effectively restores the heterogeneous profiles of intensity and geometry of moving objects against the complex background interferences. Experiments show that the proposed method significantly outperforms state-of-the-art methods in accurate moving contrast-filled vessel extraction with excellent flexibility and computational efficiency.

show abstract

A deep joint sparse non-negative matrix factorization framework for identifying the common and subject-specific functional units of tongue motion during speech

Cited by 11 publications

References 43 publications

Synthesizing audio from tongue motion during speech using tagged MRI via transformer

Synthesizing audio from tongue motion during speech using tagged MRI via transformer

Speech motion anomaly detection via cross-modal translation of 4D motion fields from tagged MRI

Working memory inspired hierarchical video decomposition with transformative representations

Contact Info

Product

Resources

About