Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix best_dev_checkpoint path #167

Open
wasertech opened this issue Jan 29, 2023 · 0 comments
Open

Fix best_dev_checkpoint path #167

wasertech opened this issue Jan 29, 2023 · 0 comments

Comments

@wasertech
Copy link

wasertech commented Jan 29, 2023

Produced checkpoints are using absolute path where they should really be using relative ones.

Checkpoints FR STT 0.9

# ./best_dev_checkpoint
model_checkpoint_path: "/mnt/checkpoints/best_dev-221133"
all_model_checkpoint_paths: "/mnt/checkpoints/best_dev-221133"

Here is English checkpoint for reference.

# ./best_dev_checkpoint
model_checkpoint_path: "best_dev-3663881"
all_model_checkpoint_paths: "best_dev-3663881"

Fix should probably go into package.sh I think.

Related : coqui-ai/STT#2338
^ STT's lm_optimizer throws a NotFoundError from TF, since the path points to nowhere in transfer-learning configuration, as checkpoint path is /transfer-checkpoint/ instead of /mnt/checkpoints/.

To use an existing checkpoint, just ensure the `docker run` includes a mount such as:
`type=bind,src=PATH/TO/CHECKPOINTS,dst=/transfer-checkpoint`. Upon running, data
will be copied from that place.

if [ -f "/transfer-checkpoint/checkpoint" -a ! -f "/mnt/models/output_graph.pb" ]; then
echo "Using checkpoint from ${TRANSFER_CHECKPOINT}"
cp -a /transfer-checkpoint/* /mnt/checkpoints/
fi;

That was for DS but with STT, we have access to --load_checkpoint_dir and --save_checkpoint_dir flags instead.

	if [ -f "/transfer-checkpoint/checkpoint" -a ! -f "/mnt/models/output_graph.tflite" ]; then
		echo "Using checkpoint from ${TRANSFER_CHECKPOINT}"
		# use --load_checkpoint_dir for transfer learning
		LOAD_CHECKPOINT_FROM="--load_checkpoint_dir /transfer-checkpoint --save_checkpoint_dir /mnt/checkpoints"
	else
		LOAD_CHECKPOINT_FROM="--checkpoint_dir /mnt/checkpoints/"
	fi;

Example from my branch stt140-cv9

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant