Send data or custom fields from streamText to useChat similar to annotations

[jakesteelman]

Feature Description

It would be awesome if we had a similar mechanism to message annotations but specifically for arbitrary data such as the model used to generate a response, the name/icon/id of the assistant used (not OpenAI assistants, just a custom one if your platform uses them) or the token usages. I am currently using annotations for this use case, but this feels like an antipattern - more in the "additional context" section of this issue.

On the server, you would instantiate a StreamData object and when you want to add/modify data that gets streamed back to the client, you call a method like...

const data = new StreamData();
data.appendMessageData({ data: { some: "json" }})

...then on the client via useChat you would intercept the data with...

const {
    // ...
} = useChat({
    // ...
    onFinish: async (message) => {
        const messageData = message.data 
        console.log('Message Data', message.data) // 3.3.x currently always prints undefined
    },
});

....and you could render it like this:

return (
   <div>
     messages.map(message => (
        <div>
           <span>{message.role}</span>
           <span>{message.content}</span>
           {message.data?.model && <span>Model: {message.data.model}</span>}
        </div>
     )
   </div>
)

Use Case

Feature would be useful in cases where message-level metadata unrelated to RAG citations or annotations needs to be streamed back to the client, such as the model used or the assistant used, as described above in the feature description.

For example, we have a very rudimental, custom LLM router on our server to use a "fallback" LLM in cases where the user's requested LLM is not available (rate limit, service down , connection errors, etc). As a result, the server needs to be able to communicate back to the client what LLM it actually used, and not which one was requested from the client side, since the two may be different.

Additional context

Our application requires storing some custom fields on each message, like:

• the model used to generate it
• details about the assistant the user sent the chat to (this is our own custom implementation of assistants, not to be confused with OpenAI assistants)
• various other fields like token usage.

Some of these don't need to be rendered client side (like token usage) but some do - like details about the assistant (so we can render the assistant's name and icon).

Currently, we are able to send arbitrary data to the server with no issues. We're using the append method provided by the useChat hook, and sending the data via the data attribute of Message.

append({
    id: generateId(),
    role: "user" as RoleType,
    content: input,
    createdAt: new Date(),
    experimental_attachments: attachedFiles,
    data: {
        model: 'gpt-4o',
        // ...
    }
});

Then, we can grab the data from the last message server-side and customize the response based on these data (using a different model or prompt, etc)

However, there's no way for us to receive arbitrary data from the server on a per-message basis.

I am aware of the StreamData.append() method. However, since calling that adds to a global data object, trying to use for our per-message use case seems hacky and unstable; we'd need to link the data object up with the message via some identifier, and to add the fields to the appropriate messages so we can render the fields would require modifying state an extra time.

Currently, as a workaround, we're using annotations:

StreamData.appendMessageAnnotation({ 
    data: { 
        model: "gpt-4o",
        // ...
    }
})

I'm not a huge fan of this either. I can't find much in the docs or in the repo about how annotations are supposed to be used, but from what I can gather and from my assumptions, these are supposed to be used for citing documents retrieved during a RAG chain, not for arbitrary or custom data. So it seems like an antipattern to use them for this case.

Here are some related items:

• Custom fields to the response assistant when append #688
• Message level streaming data #585
• Generic Messages #1180