The i-Treasures Intangible Cultural Heritage dataset

Grammalidis, Nikos; Dimitropoulos, Kosmas; Tsalakanidou, Filareti; Kitsikidis, Alexandros; Roussel, Pierre; Denby, B.; Chawah, Patrick; Buchman, Lise Crevier; Dupont, Stéphane; Laraba, Sohaïb; Picart, Benjamin; Tits, Mickaël; Tilmanne, Joëlle; Hadjidimitriou, Stelios; Hadjileontiadis, Leontios; Charisis, Vasileios; Volioti, Christina; Stergiaki, A.; Manitsaris, Athanasios; Bouzos, Odysseas; Manitsaris, Sotiris

doi:10.1145/2948910.2948944

Cited by 18 publications

(10 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…By means of a text presented during the announcement of the test participants have been informed before the study starts about: the general goals and duration of the online survey; the payment that can be expected when finishing the survey; and the fact that their answers will be used to establish average responses to the questions for a scientific publication on audio quality. Data Availability Statement: Datasets used in this paper: The singing dataset used in this paper was created by combining CREL Research Database (SVDB) [52], NUS sung and spoken lyrics corpus [53], the Byzantine singing from the i-Treasures Intangible Cultural Heritage dataset [54], PJS phonemebalanced Japanese singing-voice corpus [55], JVS-MuSiC [56], Tohoku Kiritan and Itako singing database [57], as well as internal singing databases used for the IRCAM Singing Synthesizer [59] and other projects. For speech we use a combined dataset of VCTK [50] and Att-HACK [51].…”

Section: Future Workmentioning

confidence: 99%

A Bottleneck Auto-Encoder for F0 Transformations on Speech and Singing Voice

Bous

Röebel

2022

Information

View full text Add to dashboard Cite

In this publication, we present a deep learning-based method to transform the f0 in speech and singing voice recordings. f0 transformation is performed by training an auto-encoder on the voice signal’s mel-spectrogram and conditioning the auto-encoder on the f0. Inspired by AutoVC/F0, we apply an information bottleneck to it to disentangle the f0 from its latent code. The resulting model successfully applies the desired f0 to the input mel-spectrograms and adapts the speaker identity when necessary, e.g., if the requested f0 falls out of the range of the source speaker/singer. Using the mean f0 error in the transformed mel-spectrograms, we define a disentanglement measure and perform a study over the required bottleneck size. The study reveals that to remove the f0 from the auto-encoder’s latent code, the bottleneck size should be smaller than four for singing and smaller than nine for speech. Through a perceptive test, we compare the audio quality of the proposed auto-encoder to f0 transformations obtained with a classical vocoder. The perceptive test confirms that the audio quality is better for the auto-encoder than for the classical vocoder. Finally, a visual analysis of the latent code for the two-dimensional case is carried out. We observe that the auto-encoder encodes phonemes as repeated discontinuous temporal gestures within the latent code.

show abstract

Section: Future Workmentioning

confidence: 99%

A Bottleneck Auto-Encoder for F0 Transformations on Speech and Singing Voice

Bous

Röebel

2022

Information

View full text Add to dashboard Cite

show abstract

“…In (12), w represents the weight, b corresponds to the deviation, and h t−1 denotes the output value corresponding to the time t − 1. Meanwhile, x t refers to the input value, σ represents the activation function, and f stands for the forget gate.…”

Section: Skeleton Movement Recognition Based On Optimizedmentioning

confidence: 99%

The Fusion Application of Deep Learning Biological Image Visualization Technology and Human-Computer Interaction Intelligent Robot in Dance Movements

Jin

Wen

Xie

2022

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

The paper aims to apply the deep learning-based image visualization technology to extract, recognize, and analyze human skeleton movements and evaluate the effect of the deep learning-based human-computer interaction (HCI) system. Dance education is researched. Firstly, the Visual Geometry Group Network (VGGNet) is optimized using Convolutional Neural Network (CNN). Then, the VGGNet extracts the human skeleton movements in the OpenPose database. Secondly, the Long Short-Term Memory (LSTM) network is optimized and recognizes human skeleton movements. Finally, an HCI system for dance education is designed based on the extraction and recognition methods of human skeleton movements. Results demonstrate that the highest extraction accuracy is 96%, and the average recognition accuracy of different dance movements is stable. The effectiveness of the proposed model is verified. The recognition accuracy of the optimized F-Multiple LSTMs is increased to 88.9%, suitable for recognizing human skeleton movements. The dance education HCI system’s interactive accuracy built by deep learning-based visualization technology reaches 92%; the overall response time is distributed between 5.1 s and 5.9 s. Hence, the proposed model has excellent instantaneity. Therefore, the deep learning-based image visualization technology has enormous potential in human movement recognition, and combining deep learning and HCI plays a significant role.

show abstract

“…All experiments are trained on the same dataset as [16], which is a combined dataset of CREL Research Database (SVDB) [18], NUS sung and spoken lyrics corpus [19], from the i-Treasures Intangible Cultural Heritage dataset [20] PJS phoneme-balanced Japanese singing-voice corpus [21], JVS-MuSiC [22], Tohoku Kiritan and Itako singing database [23], VocalSet: A singing voice dataset [24], as well as singing recordings from our internal singing databases used for the IR-CAM Singing Synthesizer [25] and other projects.…”

Section: Datamentioning

confidence: 99%

Analysis and transformations of intensity in singing voice

Bous¹,

Röebel²

2022

Preprint

View full text Add to dashboard Cite

In this paper we introduce a neural auto-encoder that transforms the voice intensity in recordings of singing voice. Since most recordings of singing voice are not annotated with voice intensity we propose a means to estimate the relative voice intensity from the signal's timbre using a neural intensity estimator. Two methods to overcome the unknown recording factor that relates voice intensity to recorded signal power are given: The unknown recording factor can either be learned alongside the weights of the intensity estimator, or a special loss function based on the scalar product can be used to only match the intensity contour of the recorded signal's power. The intensity models are used to condition a previously introduced bottleneck auto-encoder that disentangles its input, the mel-spectrogram, from the intensity. We evaluate the intensity models by their consistency and by their fitness to provide useful information to the auto-encoder. A perceptive test is carried out that evaluates the perceived intensity change in transformed recordings and the synthesis quality. The perceptive test confirms that changing the conditional input changes the perceived intensity accordingly thus suggesting that the proposed intensity models encode information about the voice intensity.

show abstract

The i-Treasures Intangible Cultural Heritage dataset

Cited by 18 publications

References 3 publications

A Bottleneck Auto-Encoder for F0 Transformations on Speech and Singing Voice

A Bottleneck Auto-Encoder for F0 Transformations on Speech and Singing Voice

The Fusion Application of Deep Learning Biological Image Visualization Technology and Human-Computer Interaction Intelligent Robot in Dance Movements

Analysis and transformations of intensity in singing voice

Contact Info

Product

Resources

About