VALL-E can reproduce any voice based on a three-second audio sample.

Trained on 60,000 hours of English speech data, VALL-E, which is Microsoft’s (NASDAQ: MSFT) new AI tool, can replicate any voice based on its analysis of a three-second audio sample. In an advancement of this technology, VALL-E can also simulate the tone and emotions of the speaker. (English Jagran)

Here is an extract from Cornell University’s research paper on VALL-E:

“Experiment results show that Vall-E significantly outperforms the state-of-the-art zero-shot TTS system [AI that recreates voices it’s never heard] in terms of speech naturalness and speaker similarity. In addition, we find VALL-E could preserve the speaker’s emotion and acoustic environment of the acoustic prompt in synthesis.”

How it works

VALL-E analyzes a person’s voice by breaking it down into components and then using its training to synthesize how the voice would sound if it were speaking different phrases.

It builds on EnCodec, a technology developed by Meta (NASDAQ: META) in October 2022.

Its applications include speech editing, text-to-voice tool, and (in combination with other generative AI tools such as GPT-3) to build an audio creation system.

However, there are ethical concerns surrounding the use of the tool. It could be used by hackers to impersonate a family member’s voice to request (and receive) confidential data. Or it could masquerade as an eminent personality such as a politician. Another risk, it could be used to fool systems that depend on voice recognition for identification.

Related Story:  Microsoft Mulling $10B Investment in ChatGPT Creator OpenAI

Image by Gordon Johnson from Pixabay