Prompt errors for Llama3.1

[el-hash-1]

Using the default code on huggingface page link, the prompt generated for tool calling is incorrect.
This v3.2 model is based on Llama3.1 and it should be using this template defined in your repo

But this is the model output for the default tool and message on hf

Can someone please fix this?

And if you could provide a huggingface inference just like llama_cpp example that would be great!

[jeffrey-fong]

As mentioned here, we provide custom code to parse the raw response into JSON responses. Feel free to disable it by removing trust_remote_code=True when initializing your model and you will get the raw model responses.

[el-hash-1]

As mentioned here, we provide custom code to parse the raw response into JSON responses. Feel free to disable it by removing trust_remote_code=True when initializing your model and you will get the raw model responses.

That's not what i meant, I am not just talking about the generated text, let me rephrase it.

The functionary-small-v3.2 model has the prompt format different from what it is supposed to be for llama3.1, If you check your tests/prompt_test_v3-llama3.1.txt

This is how it should look like if your base model is llama3.1 but it seems like the base model for your 3.2 version model is llama3 instead because the prompt matches to the tests/prompt_test_v3.llama3.txt

[jeffrey-fong]

v3.2 is finetuned with the prompt template in tests/prompt_test_v3.llama3.txt, not tests/prompt_test_v3-llama3.1.txt. The base model is llama3.1, not llama3. Llama3.1 can be finetuned using a different prompt template.