Effect CommunityEC
Effect Community•6mo ago
marbemac

Making `prompt` Optional in `AiChat.streamText` for Continuation Use Cases

Thoughts on making prompt optional on AiChat.streamText (or enabling the following use case in some other way)? Sometimes you just want the LLM to continue where it left off, along the lines of the example below. The first LLM call finishes with a stop reason of tool-calls, and the loop just needs to call the llm again w the history so that it can complete the task requested in the prompt. I'm passing prompt: '' atm, but this results in empty text parts, which providers like Anthropic reject.

const runPrompt = Effect.fn(function* ({ prompt }: { prompt: AiInput.Raw }) {
  const chat = yield* AiChat.AiChat;

  const result = yield* chat
    .streamText({
      prompt,
      toolkit: DemoToolkit,
    })
    .pipe(Stream.runCollect);

  return Option.match(result.pipe(Chunk.last), {
    onSome: result => result.finishReason,
    onNone: () => 'unknown' as const,
  });
});

const Gpt4o = OpenAiLanguageModel.model('gpt-4.1');

const main = Effect.gen(function* () {
  const prompt = `Call the ${GetRandomNumber.name} tool with min of 3 and max of 10, and then generate that many jokes.`;

  const chat = yield* AiChat.fromPrompt({ prompt });

  let finishReason: FinishReason | undefined;
  
  // passing empty prompt atm
  const runner = runPrompt({ prompt: '' });

  yield* Effect.repeat(
    Effect.gen(function* () {
      finishReason = yield* runner;
    }),
    {
      while: () => finishReason === undefined || finishReason === 'tool-calls',
    },
  ).pipe(Effect.provideService(AiChat.AiChat, chat));
}).pipe(Effect.provide(Gpt4o));
Was this page helpful?