Vocal Isolation and Voice Conversion: How AI Covers Actually Work Under the Hood

By Tania Alwan

I'm a music blogger, not an ML engineer, but I've spent enough time running AI cover pipelines (and reading the repos behind them) that I wanted to write down how the pieces fit together. If you've ever wondered what happens between "upload a song" and "hear it sung by a different voice", this is the plain-English version, with pointers to the open-source projects you can run yourself.

An AI cover pipeline has two very different ML problems chained together:

1. Vocal isolation (music source separation): pull the voice out of a mixed track.
2. Singing voice conversion: change who is singing while keeping what is sung.

Then there's a boring but important third stage: put the converted vocal back on the instrumental.

Part 1: Vocal isolation

The problem

A mixed song is a single waveform, the sum of every instrument. Vocal isolation asks: given that sum, estimate the vocal component. It's an underdetermined problem, since infinitely many combinations of sources could add up to the same mix. Models solve it by learning priors from thousands of multitrack recordings where the true stems are known.

Spectrogram masking

The classic approach works in the time-frequency domain:

1. Compute a short-time Fourier transform (STFT) of the mix to get a spectrogram.
2. A neural network predicts a mask: for each time-frequency bin, a value between 0 and 1 saying how much of that energy belongs to the vocal.
3. Multiply the mask by the mix spectrogram and invert the STFT, usually reusing the mixture's phase.

Deezer's Spleeter (2019) popularised this with a U-Net over spectrograms and pretrained 2-, 4- and 5-stem models. It's fast and still useful, but reusing the mixture phase is one reason older separators sound "phasey" or watery.

Waveform and hybrid models

Demucs, from Meta's research team, started as a waveform-domain model: an encoder-decoder that takes raw audio in and outputs raw audio per source, so it learns phase implicitly. Later versions went hybrid, combining a spectrogram branch and a waveform branch, and the newest variant (HT Demucs) adds a transformer between them. The standard models output four stems (drums, bass, other, vocals), and there's an experimental six-stem model that adds guitar and piano.

Trying it locally is about two commands:

pip install demucs demucs --two-stems=vocals my_song.wav 

--two-stems=vocals gives you vocals.wav and no_vocals.wav, which is all a cover pipeline needs.

If you prefer a GUI, Ultimate Vocal Remover (UVR) is a free open-source desktop app that wraps several model families (MDX-Net, Demucs, VR architecture) and lets you ensemble them.

Why separation fails

Knowing the failure modes saves you a lot of wasted compute downstream:

• Spectral overlap. Distorted guitars and synth pads share frequency ranges with the voice, so the mask can't cleanly assign energy.
• Reverb and delay. A wet vocal's tail is smeared in time. Part of it ends up in the instrumental, part stays on the vocal as a washy residue.
• Backing vocals. Most models treat all voices as one "vocals" source, so harmonies come along with the lead.
• Lossy sources. Low-bitrate codecs already threw away high-frequency detail, and the model has nothing to reconstruct from.

That's why a practical pipeline usually adds a cleanup stage after separation: dereverberation, echo removal and denoising. These are often separate models trained specifically for each artefact.

Part 2: Singing voice conversion

The core idea: disentangle content from timbre

Voice conversion tries to split a vocal into:

• Content: phonetic information, i.e. what is being sung.
• Pitch: the fundamental frequency (F0) contour, i.e. the melody.
• Timbre: speaker identity, i.e. who is singing.

Then it swaps the timbre and resynthesises audio.

Content features from self-supervised models

Modern open-source systems don't use hand-made phoneme recognisers. They use self-supervised speech models such as HuBERT or ContentVec, which turn audio into frame-level embeddings that capture linguistic content. ContentVec in particular was designed to reduce how much speaker identity leaks into those embeddings, which is exactly what conversion needs.

Pitch extraction

Singing needs accurate F0, otherwise the converted voice warbles or jumps octaves. Pipelines use pitch trackers such as CREPE, Harvest or RMVPE (a model designed to estimate vocal pitch even from polyphonic music). Most tools let you shift the extracted pitch by a number of semitones, which is how you move a song into a target voice's comfortable range, for example when converting a male vocal to a female voice model.

The two open-source families you'll meet

so-vits-svc combines a SoftVC content encoder with a VITS-style decoder, a model architecture that pairs a variational autoencoder with adversarial training and a neural vocoder to produce waveforms directly.

RVC (Retrieval-based Voice Conversion) builds on similar VITS-style synthesis but adds a retrieval trick: during training it stores content features from the target singer in an index (using Faiss). At inference, each incoming content frame is blended with its nearest neighbours from that index. Pulling the input features toward real features from the target voice reduces timbre leakage from the source singer. An "index rate" parameter controls how strong that blending is.

Training an RVC voice model needs only minutes of clean, dry audio of the target voice, which is why so many community models exist. Data quality matters far more than quantity: reverb, music bleed or noise in the training set gets baked into the voice.

Why conversion fails

• Bleed in the input. The content encoder dutifully encodes hi-hat transients as "content", and the decoder tries to sing them.
• Range mismatch. Asking a voice model to sing far outside the range it was trained on produces strain and artefacts. Transpose instead.
• Breaths and fricatives. Unvoiced sounds have no pitch, and some pipelines handle them poorly. Many tools offer a "protect" setting for consonants and breaths.

Part 3: Putting it together

A full pipeline looks like this:

mix.wav └─► separation (Demucs / MDX) ─► vocals.wav + instrumental.wav └─► cleanup (de-reverb, de-echo, denoise) ─► vocals_dry.wav └─► F0 extraction + content encoding └─► voice conversion (RVC / so-vits-svc) ─► vocals_converted.wav └─► mix with instrumental (+ new reverb) ─► cover.wav 

Running this yourself is a great learning project, but it means managing GPUs, model checkpoints and a few Python environments that don't always agree on dependency versions.

The hosted route

If you'd rather not maintain that stack, hosted tools wrap the same stages behind a UI. I've been using Voicestars for this, mainly because each stage is exposed as its own tool, which maps nicely onto the pipeline above:

• Separation: the AI stems splitter outputs vocals, drums, bass, guitars, piano and "other".
• Cleanup: separate reverb remover, echo remover and noise remover tools.
• Conversion: a library of AI voices, or training a model on your own voice. The page recommends around 10 minutes of clean audio and says training typically takes one to two hours.

Usage is credit-based. The pricing page lists the per-feature cost, for example 50 credits per AI cover and per-minute rates for the audio tools. Voicestars doesn't publish which model architectures it uses internally, so I can't tell you whether it runs Demucs or RVC under the hood. What I can tell you is that the same input-quality rules apply: clean separation in, clean conversion out.

A note on ethics

Voice conversion makes impersonation trivially easy. If you build or use these pipelines, convert your own voice or voices you have permission to use, label AI vocals clearly, and don't publish imitations of real artists commercially without consent. I covered the legal side in more depth in this piece on whether AI covers are legal.

Further reading

• The Voicestars blog post on its stem splitter
• A practical acapella extractor guide I wrote on cleaning separated vocals
• The Demucs, Ultimate Vocal Remover, so-vits-svc and RVC repositories on GitHub, whose READMEs are the best primary sources

Try it

Run demucs --two-stems=vocals on a track you own, listen to the vocal stem solo, then push the same file through the Voicestars stems splitter and compare. Hearing where each one leaks is the fastest way to understand what these models are really doing.

Disclosure: this article was written in partnership with Voicestars.