Effect CommunityEC
Effect Community•3mo ago•
25 replies
Frajer

Handling LLM Streaming with Effect and Streams

Hey guys,

I have managed to make effect rpc work with effect atom.

But I think my mental model around Streams is broken. The following example works fine and I somewhat get it.

But how can I handle for example and LLM streaming in text? Do I need to constantly pull it? or is there a better way to go about this? Should I even be using atom for this?

'use client';
import { FetchHttpClient } from '@effect/platform';
import { RpcClient, RpcSerialization } from '@effect/rpc';
import { Cause, Layer } from 'effect';
import { UserRpcs } from '@/lib/request';

import { AtomRpc, Result, useAtom } from '@effect-atom/atom-react';

// Choose which protocol to use
const ProtocolLive = RpcClient.layerProtocolHttp({
  url: 'http://localhost:3000/api/rpc',
}).pipe(
  Layer.provide([
    // use fetch for http requests
    FetchHttpClient.layer,
    // use ndjson for serialization
    RpcSerialization.layerNdjson,
  ])
);

class UserClient extends AtomRpc.Tag<UserClient>()('UserClient', {
  group: UserRpcs,
  protocol: ProtocolLive,
}) {}

export default function UserListComponent() {
  const [result, pull] = useAtom(UserClient.query('UserList', void 0));

  console.log(result);
  return Result.match(result, {
    onInitial: () => <div>Loading...</div>,
    onFailure: error => <div>Error: {Cause.pretty(error.cause)}</div>,
    onSuccess: success => (
      <div>
        <ul>
          {success.value.items.map(user => (
            <li key={user.id}>{user.name}</li>
          ))}
        </ul>
        <button onClick={() => pull()}>Load more</button>
        {success.waiting ? <p>Loading more...</p> : <p>Loaded chunk</p>}
      </div>
    ),
  });
}
Was this page helpful?