PE-wav2vec: A Prosody-Enhanced Speech Model for Self-Supervised Prosody Learning in TTS
Authors: Zhao-Ci Liu, Liping Chen, Ya-Jun Hu, Zhen-Hua Ling, Senior Member, IEEE, and Jia Pan
Abstract: This paper investigates leveraging large-scale untranscribed speech data to enhance the prosody modeling capability of text-to-speech (TTS) models. Based on the self-supervised speech model wav2vec 2.0, the Prosody-Enhanced wav2vec (PE-wav2vec) is proposed by introducing prosody learning. Specifically, prosody learning is achieved by applying supervision from the linear predictive coding (LPC) residual signals on the initial Transformer blocks in the wav2vec 2.0 architecture. The embedding vectors extracted with the initial Transformer blocks of the PE-wav2vec model are utilized as prosodic representations for the corresponding frames in a speech utterance. To apply the PE-wav2vec representations in TTS, an acoustic model named Speech Synthesis model conditioned on Self-Supervisedly Learnt Prosodic Representations (S4LPR) is designed based on FastSpeech 2. The experimental results demonstrate that the proposed PE-wav2vec model is capable of providing richer prosody descriptions of speech than the vanilla wav2vec 2.0 model. Furthermore, the S4LPR model using PE-wav2vec representations can effectively improve the subjective naturalness and reduce the objective distortions of synthetic speech when compared to baseline models.
Translation: The corridor inside is spacious and gloomy, with green shaded lamps overhead. The carpet below is so thick that even if you dropped a firecracker on it, it wouldn't make a sound.
Translation: The seal was smashed open, and the Buddha's head rolled out from inside, tumbling down the pile of books and landing on the concrete floor with a thud.
Translation: Therefore, Mahakasyapa, at all costs, resorted to a strategy of feigning weakness and enduring attacks, holding out until the most critical moment, before finally launching a surprise attack to secure victory in one strike.
Translation: When Zeng Sitao had these thoughts in his mind, he couldn't help but curse himself for being ridiculous, realizing that his level of wild imagination had recently taken a significant leap.
Translation: You see, the dividends of an industry are often reaped not by those with the highest rank or position, but by those who are most skilled in writing and speaking. That's where the truth lies.
Translation: He has the opportunity to make money right in front of him. Asking him to give up this opportunity for some long-term vision is essentially asking him to forgo his current means of survival, which is not very considerate.
Translation: So, when is there an opportunity to break the rules? There are three situations. The first is when the cost of connection is already very low. For example, at home, the rules between a couple are not easy to maintain for a long time. This is the so-called "it is difficult for an honest official to manage household matters."
Translation: The company naturally handles large-scale shipments. They excel at making mooncake purchases cheaper and shipping more efficient, focusing on efficiency-oriented tasks.
Translation: You see, pursuing efficiency is certainly important, but it has a downside: you may only be doing things that play to your strengths. This can create a cognitive barrier, leading you to mistakenly believe that you are working hard and being effective.
Translation: The corridor inside is spacious and gloomy, with green shaded lamps overhead. The carpet below is so thick that even if you dropped a firecracker on it, it wouldn't make a sound.
Translation: The seal was smashed open, and the Buddha's head rolled out from inside, tumbling down the pile of books and landing on the concrete floor with a thud.
Translation: Therefore, Mahakasyapa, at all costs, resorted to a strategy of feigning weakness and enduring attacks, holding out until the most critical moment, before finally launching a surprise attack to secure victory in one strike.
Translation: When Zeng Sitao had these thoughts in his mind, he couldn't help but curse himself for being ridiculous, realizing that his level of wild imagination had recently taken a significant leap.