The model is about the question of time

[kaisersama112]

Thanks to the author for providing this model, after local testing, the model does have a strong ability to invoke tools, but - the model has a very fatal problem (about the derivation of time)

[musabgultekin]

Hi, unfortunately we have not trained the model with non-english conversations :(

[kaisersama112]

您好，不幸的是我们还没有用非英语对话训练模型:(

It doesn't really matter about the language, but I would like to ask if there is a way to improve the model's ability to understand the reasoning of time, or whether you will improve the model's ability to do this in the next version

[kaisersama112]

您好，不幸的是我们没有有用的非英语对话训练模型:(

For example: today, this week, this month, this vague correct interpretation of time

[jeffreymeetkai]

Hi, we did not train the the current models (v2.4) on a variety of system prompts so it may not understand and follow your system prompts that well. We are currently looking at improving future models in this aspect. Actually, you can consider using our code generation feature to reason about time. Here's a simple example:

from IPython.core.interactiveshell import InteractiveShell
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="functionary")
ipython = InteractiveShell.instance()

messages = [
    {"role": "user", "content": "Hello"},
    {"role": "assistant", "content": "Hello! How can I assist you today?"},
    {"role": "user", "content": "What is the date for this past monday?"},
]

output = client.chat.completions.create(
    model="meetkai/functionary-small-v2.4",
    messages=messages,
    tools=[{"type": "code_interpreter"}],
    temperature=0.0,
)

if (
    output.choices[0].message.tool_calls is not None
    and len(output.choices[0].message.tool_calls) > 0
):
    messages.append(output.choices[0].message)
    func_output = ipython.run_cell(
        output.choices[0].message.tool_calls[0].function.arguments
    )
    messages.append(
        {
            "role": "tool",
            "tool_call_id": output.choices[0].message.tool_calls[0].id,
            "name": output.choices[0].message.tool_calls[0].function.name,
            "content": str(func_output.result),
        }
    )

output = client.chat.completions.create(
    model="meetkai/functionary-small-v2.4",
    messages=messages,
    tools=[{"type": "code_interpreter"}],
    temperature=0.0,
)

print(output.choices[0].message)

and the result is:

ChatCompletionMessage(content='The date for the past Monday is April 22, 2024.', role='assistant', function_call=None, tool_calls=[], tool_call_id=None, name=None)

In the messages, there will be an assistant tool call to the code interpreter to get the corresponding date.

However, this may not work if you provide a different date and time from the current datetime via system prompts due to the reason stated above.

It works with basic Chinese too, although we do not guarantee that it will work well as we did not train the model with non-english conversations.

messages = [
    {"role": "user", "content": "你好"},
    {"role": "assistant", "content": "你好！我可以为你提供帮助吗？"},
    {"role": "user", "content": "这周一是几月几日"},
]

ChatCompletionMessage(content='这周一是4月29日。', role='assistant', function_call=None, tool_calls=[], tool_call_id=None, name=None)

[kaisersama112]

Ok thanks a lot for the ideas provided

[sashokbg]

Hello, @kaisersama112 I have had the same problem where the model works perfectly for the functions calling part but falls a bit short on other types of inference logic.

I would say that this is normal and that we cannot have models that do everything at once. My solution was to actually implement a multi-step pipeline where multiple models will do different jobs.

For instance I have created specialized multi-shot prompt using LMQL and llama 3 just for the task of properly understanding dates and time periods. The result is then passed down to functionary which can easily call the appropriate function.

I will try to push my code to a repo on github and share with you if I find some free time :)

PS: You can also check a tool called promptflow by microsoft that has this exact "multi step" idea implemented

[sashokbg]

Also another solution I tried and that worked kinda ok was to implement a function for date picking. This way whenever the model thinks that the user needs to provide a date it will call a "choose_date" function. I then intercept this and show a datepicker on the screen:

[kaisersama112]

此外，我尝试了另一种解决方案，并且效果还不错，那就是实现日期选择功能。这样，每当模型认为用户需要提供日期时，它就会调用“choose_date”函数。然后我截获它并在屏幕上显示一个日期选择器：

Hello, thank you very much for providing ideas and methods, for the problem of time, at present I have solved it through external methods (Chinese natural language time parsing into standard time format) (Figure 1), but at present, I have encountered a new problem for complex task process models can not be effectively executed (Figure 2), I have conceived a new scheme here, the subtask is broken down and executed, the subtask interacts with the user, and the overall process has been passed through (Figure 3，Figure 4), But there is no good way to combine it with the LLM model, do you have a good idea here?