I am trying to create an API endpoint using Nextjs 13's new Route Handler solution. This API uses LangChain, and streams the response back to the frontend. When calling the OpenAI wrapper class, I am passing in the Streaming property, and supplying the callback function. This callback function then provides the stream as chunks (ie. tokens). I want to stream these tokens to the frontend to output the AI's response as it's being generated.

I was able to get this working using the "old" API route solution with the following code:

import { OpenAI } from "langchain/llms/openai";

export default async function handler(req, res) {
  const chat = new OpenAI({
    modelName: "gpt-3.5-turbo",
    streaming: true,
    callbacks: [
      {
        handleLLMNewToken(token) {
          res.write(token);
        },
      },
    ],
  });

  await chat.call("Write me a song about sparkling water.");

  res.end();
}

I am trying to convert this code to the new Route Handler solution, but I haven't been able to get this working.

I have tried many different approaches to this, with no luck.

For example:

import { NextResponse } from "next/server";

import { OpenAI } from "langchain/llms/openai";

export const dynamic = "force-dynamic";
export const revalidate = true;

export async function GET(req, res) {
  const chat = new OpenAI({
    modelName: "gpt-3.5-turbo",
    streaming: true,
    callbacks: [
      {
        handleLLMNewToken(token) {
          // res.write(token);
          return new NextResponse.json(token);
        },
      },
    ],
  });

  await chat.call("Write me a song about sparkling water.");
}

There just seems to be no way to "write" the tokens to the response as they are streamed to the Route Handler's response.

Any assistance will be GREATLY appreciated.

I am trying to create an API endpoint using Nextjs 13's new Route Handler solution. This API uses LangChain, and streams the response back to the frontend. When calling the OpenAI wrapper class, I am passing in the Streaming property, and supplying the callback function. This callback function then provides the stream as chunks (ie. tokens). I want to stream these tokens to the frontend to output the AI's response as it's being generated.

I was able to get this working using the "old" API route solution with the following code:

import { OpenAI } from "langchain/llms/openai";

export default async function handler(req, res) {
  const chat = new OpenAI({
    modelName: "gpt-3.5-turbo",
    streaming: true,
    callbacks: [
      {
        handleLLMNewToken(token) {
          res.write(token);
        },
      },
    ],
  });

  await chat.call("Write me a song about sparkling water.");

  res.end();
}

I am trying to convert this code to the new Route Handler solution, but I haven't been able to get this working.

I have tried many different approaches to this, with no luck.

For example:

import { NextResponse } from "next/server";

import { OpenAI } from "langchain/llms/openai";

export const dynamic = "force-dynamic";
export const revalidate = true;

export async function GET(req, res) {
  const chat = new OpenAI({
    modelName: "gpt-3.5-turbo",
    streaming: true,
    callbacks: [
      {
        handleLLMNewToken(token) {
          // res.write(token);
          return new NextResponse.json(token);
        },
      },
    ],
  });

  await chat.call("Write me a song about sparkling water.");
}

There just seems to be no way to "write" the tokens to the response as they are streamed to the Route Handler's response.

Any assistance will be GREATLY appreciated.

• javascript
• next.js

Share Improve this question Follow edited May 21, 2023 at 8:35 Jonas 129k103103 gold badges328328 silver badges405405 bronze badges asked May 21, 2023 at 7:29 Leon van ZylLeon van Zyl 23144 silver badges55 bronze badges

Add a ment  | 

1 Answer 1

Sorted by: Reset to default 5

I think I might have a solution.

In the Route Handler, I create a new stream object using the TransformStream class. I then write the tokens to this stream object as they are generated. Because the stream expects bytes to be transferred to it, I use the TextEncoder to encode the token to a Uint8Array value.

Lastly, I then return this readable property of the stream in our API response. This seems to do the trick, although slightly more plex than the solution from the older API route approach.

import { OpenAI } from "langchain/llms/openai";

export const dynamic = "force-dynamic";
export const revalidate = true;

async function runLLMChain() {
  // Create encoding to convert token (string) to Uint8Array
  const encoder = new TextEncoder();

  // Create a TransformStream for writing the response as the tokens as generated
  const stream = new TransformStream();
  const writer = stream.writable.getWriter();

  const chat = new OpenAI({
    modelName: "gpt-3.5-turbo",
    streaming: true,
    callbacks: [
      {
        async handleLLMNewToken(token) {
          await writer.ready;
          await writer.write(encoder.encode(`${token}`));
        },
        async handleLLMEnd() {
          await writer.ready;
          await writer.close();
        },
      },
    ],
  });
  chat.call("Write me a song about sparkling water.");

  // Return the readable stream
  return stream.readable;
}

export async function GET(req) {
  const stream = runLLMChain();
  return new Response(await stream);
}