Is best-model always better than latest checkpoint ?

[Ca-ressemble-a-du-fake]

Hi,

I am continuing the training that I started from scratch with my own short dataset (500 samples). A best model was created at step 10548 (during first training from scratch) and since then it is the only best model although it is now at step 130k (training with continue flag).

Checkpoint 130k sounds better yet, so why a new "best_model" has not been created since the one from step 10k ?

Thanks

[erogol]

The best model is only the best in losses. So some other model might sound better.

[eginhard]

@abraham-pgagi Both are saved automatically, the latest ones with filenames checkpoint_*.pth. How often, depends on your save_step config setting.

[abraham-pgagi]

Oh ok. Thank you.

[abraham-pgagi]

@eginhard
Hi. I am finetuning VITS model.

These are the parameters of my config file (Default values): (I was not able to find the definitions for these parameters in the documentation)

    "save_step": 10000,
    "save_n_checkpoints": 5,
    "save_checkpoints": true,
    "save_all_best": false,
    "save_best_after": 0,

But I could not find where checkpoint_*.pth is stored in Drive.
I can only see two .pth files: best_model.pth and best_model=1002421.pth

For example my model is finetuning and in tensorboard it is showing step 1003165, the voice is good, but I am not seeing that corresponding model in drive.

[eginhard]

I assume you're finetuning a model that was trained for 1000000 steps. So you've so far trained 3165 steps yourself and the first regular checkpoint is only saved after 10000 steps. So either train for longer or reduce the save_step value. The parameters are defined here: https://github.com/idiap/coqui-ai-Trainer/blob/8025277823e304f124446f612782771d6704bc06/trainer/config.py#L138

[abraham-pgagi]

Thank you for sharing this.