I'm learning .NET maui by implement a local chat app using phi-4 onnx model. I have finished the basic chat flow, every thing works, but currently the bot message is appeared in one time, I want to output the response char by char like ChatGPT, need help on this.

Here are my demo code:

    // handle user input
    private async void OnSendMessage(object sender, EventArgs e)
    {
        var messageText = MessageEntry.Text?.Trim();
        if (string.IsNullOrEmpty(messageText))
            return;

        var userMessage = new ChatMessage
        {
            Id = Guid.NewGuid().ToString(),
            Text = messageText,
            IsUser = true,
            Timestamp = DateTime.Now
        };

        Messages.Add(userMessage);

        MessageEntry.Text = string.Empty;

        await HandlePhi4Message(messageText);
    }

// invoke phi4 model to generate output
private async Task HandlePhi4Message(string userQ)
{
    var systemPrompt = "You are an AI assistant that helps people find information. Answer questions using a direct style. Do not share more information that the requested by the users.";

    var fullPrompt = $"<|system|>{systemPrompt}<|end|><|user|>{userQ}<|end|><|assistant|>";
    var tokens = tokenizer.Encode(fullPrompt);

    var generatorParams = new GeneratorParams(model);

    generatorParams.SetSearchOption("max_length", 2048);
    generatorParams.SetSearchOption("past_present_share_buffer", false);
    using var tokenizerStream = tokenizer.CreateStream();
    var generator = new Generator(model, generatorParams);

    generator.AppendTokens(tokens[0].ToArray());

    var responseGuid = Guid.NewGuid().ToString();
    var botMessage = new ChatMessage
    {
        Id = responseGuid,
        Text = "",
        IsUser = false,
        Timestamp = DateTime.Now
    };
    Messages.Add(botMessage);

    while (!generator.IsDone())
    {
        generator.GenerateNextToken();
        var output = tokenizerStream.Decode(generator.GetSequence(0)[^1]);
        Console.Write(output);
        var message = Messages.FirstOrDefault(m => m.Id == responseGuid);
        if (message != null)
        {
            message.Text += output;
        }
    }

    await Task.CompletedTask;
}

I'm pretty sure in Messages.Add(botMessage); the UI didn't update, it updated after the while loop.

I found a blog post from syncfusion team:

But according to the screenshot at the end of the article, I think it's not stream response, because the AI response appeared in one time.

Below are my debug screenshots:

According to the above debug screenshots, even though Messages updated (Count is 2), but the UI didn't update, only after OnSendMessage returned, the UI get updated.

The xaml part is simple

<Button Grid.Column="1"
    Text="Send"
    Margin="5"
    CornerRadius="5"
    Clicked="OnSendMessage"
    BackgroundColor="{DynamicResource PrimaryColor}"
    TextColor="White" />

why the UI has to be updated after OnSendMessage?

---

update: Changing to await Task.Yield()

Sadly Task.Yield() not working, according the debug screenshot, the output is Phi which means the AI already generated some output, now Message count is 2, the UI didn't update, my input text introduce yourself is not cleared, even though I have MessageEntry.Text = string.Empty; before the AI operations.

I'm learning .NET maui by implement a local chat app using phi-4 onnx model. I have finished the basic chat flow, every thing works, but currently the bot message is appeared in one time, I want to output the response char by char like ChatGPT, need help on this.

Here are my demo code:

    // handle user input
    private async void OnSendMessage(object sender, EventArgs e)
    {
        var messageText = MessageEntry.Text?.Trim();
        if (string.IsNullOrEmpty(messageText))
            return;

        var userMessage = new ChatMessage
        {
            Id = Guid.NewGuid().ToString(),
            Text = messageText,
            IsUser = true,
            Timestamp = DateTime.Now
        };

        Messages.Add(userMessage);

        MessageEntry.Text = string.Empty;

        await HandlePhi4Message(messageText);
    }

// invoke phi4 model to generate output
private async Task HandlePhi4Message(string userQ)
{
    var systemPrompt = "You are an AI assistant that helps people find information. Answer questions using a direct style. Do not share more information that the requested by the users.";

    var fullPrompt = $"<|system|>{systemPrompt}<|end|><|user|>{userQ}<|end|><|assistant|>";
    var tokens = tokenizer.Encode(fullPrompt);

    var generatorParams = new GeneratorParams(model);

    generatorParams.SetSearchOption("max_length", 2048);
    generatorParams.SetSearchOption("past_present_share_buffer", false);
    using var tokenizerStream = tokenizer.CreateStream();
    var generator = new Generator(model, generatorParams);

    generator.AppendTokens(tokens[0].ToArray());

    var responseGuid = Guid.NewGuid().ToString();
    var botMessage = new ChatMessage
    {
        Id = responseGuid,
        Text = "",
        IsUser = false,
        Timestamp = DateTime.Now
    };
    Messages.Add(botMessage);

    while (!generator.IsDone())
    {
        generator.GenerateNextToken();
        var output = tokenizerStream.Decode(generator.GetSequence(0)[^1]);
        Console.Write(output);
        var message = Messages.FirstOrDefault(m => m.Id == responseGuid);
        if (message != null)
        {
            message.Text += output;
        }
    }

    await Task.CompletedTask;
}

I'm pretty sure in Messages.Add(botMessage); the UI didn't update, it updated after the while loop.

I found a blog post from syncfusion team: https://www.syncfusion/blogs/post/dotnet-maui-chatgpt-like-app-using-openai

But according to the screenshot at the end of the article, I think it's not stream response, because the AI response appeared in one time.

Below are my debug screenshots:

According to the above debug screenshots, even though Messages updated (Count is 2), but the UI didn't update, only after OnSendMessage returned, the UI get updated.

The xaml part is simple

<Button Grid.Column="1"
    Text="Send"
    Margin="5"
    CornerRadius="5"
    Clicked="OnSendMessage"
    BackgroundColor="{DynamicResource PrimaryColor}"
    TextColor="White" />

why the UI has to be updated after OnSendMessage?

---

update: Changing to await Task.Yield()

Sadly Task.Yield() not working, according the debug screenshot, the output is Phi which means the AI already generated some output, now Message count is 2, the UI didn't update, my input text introduce yourself is not cleared, even though I have MessageEntry.Text = string.Empty; before the AI operations.

• maui

Share Improve this question Follow edited Mar 11 at 17:17 Joey asked Mar 11 at 16:11 JoeyJoey 1,71144 gold badges1414 silver badges2020 bronze badges 4

• 1 is ChatMessage observable? – Jason Commented Mar 11 at 16:16
• Yes, it is. public ObservableCollection<ChatMessage> Messages { get; set; } the data binding is working, I just want to implement the stream response. – Joey Commented Mar 11 at 16:20
• 1 that is Messages, I asked about the ChatMessage class – Jason Commented Mar 11 at 17:25
• Yes, you find the point, my ChatMessage didn't implement INotifyPropertyChanged, after changing this, .Text change would trigger UI update. – Joey Commented Mar 11 at 18:29

Add a comment  | 

2 Answers 2

Sorted by: Reset to default 2

You need to let the GUI framework actually work, you can't just hog the main thread until it's done generating the response. Luckily that's as simple as adding a yield operation:

    while (!generator.IsDone())
    {
        generator.GenerateNextToken();
        var output = tokenizerStream.Decode(generator.GetSequence(0)[^1]);
        Console.Write(output);
        var message = Messages.FirstOrDefault(m => m.Id == responseGuid);
        if (message != null)
        {
            message.Text += output;
        }

        await Task.Yield(); // <---
    }

Also get rid of that silly await Task.CompletedTask, it's doing nothing but hide the compiler's message that was trying to help you figure it out. Plus it's embarrassing.

After some hard debugging and searching, I think I found where I am wrong.

1. I declared Messages to be ObservableCollection, but it only Add operation will be notified to update the UI, .Text field change won't, to make the UI knows that .Text field changed in Messages, I need to let my ChatMessage class implement INotifyPropertyChanged

2. After step 1, any message.Text field changed in Messages should trigger UI to update, but because of AI inference is CPU heavy, so `generator.GenerateNextToken();` will block the main thread, so I my case the UI will be updated after the AI part.

To solve my problem, I should try make `generator.GenerateNextToken()` async, so its execution won't block the main thread, I need to dig into Microsoft.ML.OnnxRuntimeGenAI to find a async method or find another library that provide async method.