In this work, we investigate the effectiveness of two techniques for improving variational autoencoder (VAE) based voice conversion (VC). First, we reconsider the relationship between vocoder features extracted using the high quality vocoders adopted in conventional VC systems, and hypothesize that the spectral features are in fact F0 dependent. Such hypothesis implies that during the conversion phase, the latent codes and the converted features in VAE based VC are in fact source F0 dependent. To this end, we propose to utilize the F0 as an additional input of the decoder. The model can learn to disentangle the latent code from the F0 and thus generates converted F0 dependent converted features. Second, to better capture temporal dependencies of the spectral features and the F0 pattern, we replace the frame wise conversion structure in the original VAE based VC framework with a fully convolutional network structure. Our experiments demonstrate that the degree of disentanglement as well as the naturalness of the converted speech are indeed improved.
Discussions and Conclusions
In this work, we investigated two approaches to improve VAE-VC. A FCN structure was applied to take sequential inputs rather than performing conversion frame by frame, thus capable of capturing the temporal relationship of speech. The F0 conditioning mechanism, motivated by a reconsideration of the relationship between vocoder features, helps eliminate residual F0 information in the latent code that might potentially harm the conversion performance. The experimental evaluations showed that the impact of FCNs on the objective measures and subjective speech naturalness assessment was positive. On the other hand, F0 conditioning showed promising results in increasing the degree of disentanglement of latent codes, and achieved high speech quality equivalent to FCN-CDVAE.
We attribute the insignificant improvement brought by the F0 conditioning scheme to a mismatch between training and conversion. The converted F0 obtained through such a simple F0 conversion process adopted in this and many past works is far from natural. As a result, in the conversion phase, the input combination which consisted of latent codes extracted from normal MCCs and the unnatural converted F0 might have not be seen by the model during training, thereby causing a degradation in quality.
While the above mentioned mismatch could be one possible reason, we would like to highlight that, although the motivation of applying F0 conditioning to VAE-VC was based on the assumption that the vocoder spectral features are F0 dependent, the design of vocoders was to separate these two features as much as possible. The amount of F0 information that resides in the spectral features might already be small enough for our proposed mechanism to eliminate. In the future, we plan to apply this general idea to rawer input features that are richer in F0, e.g., magnitude spectrograms.