VALL-E X
관심 있는 부분만 요약했습니다..
https://arxiv.org/pdf/2303.03926
conditional language modeling task with neural codec codes
AR language model 를 사용한 paired phoneme sequences 로부터 첫번째 Encodec quantizer 로 audio codec 을 만들고, 그 코덱들을 나머지 quantizer 로 병렬로 NAR model 을 이용해서 코드를 생성한다
multilingual autoregressive codec LM, multilingual non-autoregressive codec LM 이 acoustic tokens 를 서로 다르게 세부적으로 생성
acoustic quantizer, vall-e 에서처럼 Encodec, encoder-decoder model with L(8) quantization layer, 각 레이어는 75hz 에서 1024의 quantized code 생성
multilingual AR codec LM
autoregressively generates acoustic tokens,
only used to predict the acoustic tokens from the first quantizer layer of EnCodec model
from semantic tokens (phoneme sequence) -> acoustic tokens
decoder theta mar is optimized by maximizing the log-likelihood
S : the transcribed phoneme sequence
A : first-layer acoustic tokens extracted from speech X
decoder theta mar : modeling the concatenated sequence <s, a>, is trained to predict A autoregressively.
<> : sequence concatenation operation
p(.) : softmax function
multilingual NAR codec LM
non-autoregressive transformer language model aiming at iteratively generating the rest layers of acoustic tokens from the first layer
S : the phoneme sequence of the current sentence
A~ : the acoustic token sequence of the another sentence with same speaker, from the previous sentence in the dataset where the adjusted sentences are usually segmented from the same paragraph. it is expected to have the same characteristics of voice as the current sentence and is used as an additional reference for cloning the target voice
each layer l
A : the embeddings of l-1 layers' acoustic tokens, summed up layerwise as input
the learning objective for the l-layer acoustic tokens A can be calculated as (2)
multilingual training
take advantage of bilingual speech-transcription corpus, pairs of (S^s, A^s), (S^t, A^t)
Language ID Module:
to guide the speech generation for specific languages
maybe confused to select suitable acoustic tokens for the specific language since it is trained with multilingual data
different characteristics for example tones, which increases the difficulty of adjusting the speaking style across languages.
adding language info is surprisingly effective in guiding the right speaking style and relieving the L2 accent problem
we embed language ids into dense vectors and add them to the embeddings of acoustic tokens
without LID or wihout wrong LID, the translation quality decrease, while the speaker similarity between hypothesis and source speech increase.
the target LID reduces the transfer of information, which means the model without LID or with source LID will better maintain the sound of the original speaker.
foreign accent control
L2 (second-language, or foreign) accent problem, the synthesized speech sounds like the accents of a foreigner, has arisen in cross-lingual TTS systems.
adding LID can boost speech quality
by using correct LID embedding, the model trans is able to alleviate the foreign accent problem
Code-Switch Speech Synthesis
aims to produce a fluent and consistent voice for code-switch text
without special optimization for code-switch setting, the model provides a promising solution to code-switch speech synthesis
due to its strong in-context learning ability, model can synthesize fluent code-switch speech with a consistent voice.
phonemization & quantization
unified phoneme set english and chinese : https://github.com/speechio/BigCiDian
EnCodec, which employs residual vector quantization to iteratively quantize speech to a codebook according to the residual after quantization
Training details
When optimizing φMNAR, instead of accumulating all layer’s loss in Eqn. (2), we randomly select one layer at each optimization step for efficiency. < 신기하당,,,
Inference
source speech X^s 가 주어지면,
semantic encoder -> source phonemese S^s 생성
semantic decoder -> target phonemes S^t 생성
Encodec -> source acoustic token A^s
concate S^s, S^t, A^s
-> VALL-E X
-> acoustic token sequence for target speech
-> Encode decoder
-> final target speech