[Bug] XTTS v2 - short utterances finetune doesn't work

[SinanAkkoyun]

Describe the bug

I can not get short utterances (a couple words) to work without hallucinations at the end, despite my training mix being 50/50 very short and long utterances. Why won't the GPT predict the EOT token correctly if it has seen enough examples already? (1h training data at epoch 46)

Is it due to some batched training optimization that neglects EOT tokens?

To Reproduce

Finetune model on 50/50 very short and long utterances (or sometimes see with pretrained xttsv2 checkpoint and custom speaker latent) and prompt with "Program complete." or something.

Expected behavior

It should cut off after generating the sentence.

Logs

No response

Environment

{
    "CUDA": {
        "GPU": [
            "NVIDIA GeForce RTX 4090",
        ],
        "available": true,
        "version": "12.1"
    },
    "Packages": {
        "PyTorch_debug": false,
        "PyTorch_version": "2.4.0+cu121",
        "TTS": "0.24.1",
        "numpy": "1.26.2"
    },
    "System": {
        "OS": "Linux",
        "architecture": [
            "64bit",
            "ELF"
        ],
        "processor": "x86_64",
        "python": "3.11.5",
        "version": "#128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024"
    }
}

Additional context

No response

[SinanAkkoyun]

I am at epoch 241 and it just gets worse. It hallucinates, even after a 7 word sentence.
There must be something wrong with batched padding or something, I'd appreciate help.

[duringleaves]

I get hallucinations and slurs at the beginning and end of short phrases. I've trained it on a mix of short (less than 7 words) and long phrases, but it just doesn't like it.

[SinanAkkoyun]

@duringleaves you get slurs? xD

[duringleaves]

Hahaha. Well, the SLURRED words and "oy NUk!" Breakouts tamed down with a much lower temperature. Still not perfect, but more usable than I feared.