How long does it take to train Tactron2_DDC ?

[koayst]

Hi,

I have been training a "recipes/ljspeech/tacotron2-DDC" model using 1 GPU of 16GB since Aug 15 and I noticed that I am at Epoch 83/1000.

At this speed, it is going to take months to hit 1000 epoch. I didn't change any parameters found in train_tacotron2-DDC. Everything as cloned from github.

Am I doing it correctly ? I have a 4 GPU system. Should I switch to more GPU training ?

[thorstenMueller]

You do not have to train all 1000 epochs.

• Are you using GPU (CUDA) based training?
• Do you have tensorboard running?
• Are graphs looking good?
• At which step is your training process?

I've no experience with multi GPU training, but this might help improving training speed.

[koayst]

Hi @thorstenMueller,

• I am using GPU (with CUDA) to train.
• I run tensorboard just now and it was at step 53,998. I playback the audio and it sounded "robotic and broken".
• Estimate Epoch is around 133 (406 steps per epoch)
• Please see images below

I don't have much experience in this. Could you provide some guidance on what should I be looking for OR produce a video? I learnt something when I watched your youtube channel.

[carrrlosss]

@koayst have you figured out the robotic and broken sound? i'm getting the same

[koayst]

No. Didn't know what to do. Hence, give up.

[carrrlosss]

i've completed 1000 epochs over 12hrs, still sounds like a robot

[siagupta-03]

Hi @thorstenMueller,

• I am using GPU (with CUDA) to train.
• I run tensorboard just now and it was at step 53,998. I playback the audio and it sounded "robotic and broken".
• Estimate Epoch is around 133 (406 steps per epoch)
• Please see images below

I don't have much experience in this. Could you provide some guidance on what should I be looking for OR produce a video? I learnt something when I watched your youtube channel.

if not 1000 how much epochs are necessary

[thorstenMueller]

IMHO 50k steps is too low. Maybe check quality at around 100k.