Concurrent Data Processing with Pagination in Effect: Seeking Guidance
Hey guys, I want fetch items from a database and process them as fast as possible (concurrent). But the db request can return a pagination cursor.
So I actually want to fetch the first batch of items, start processing them immediately in parallel but also fetch the next batch asap and start processing them too untill no more items are left.
I read about streaming, queueing and concurrency but not sure how to tackle a technical requirement like this in Effect.
Anyone have a idea?
So I actually want to fetch the first batch of items, start processing them immediately in parallel but also fetch the next batch asap and start processing them too untill no more items are left.
I read about streaming, queueing and concurrency but not sure how to tackle a technical requirement like this in Effect.
Anyone have a idea?
Production-grade applications in TypeScript.
6,221Members
Resources
Recent Announcements
Similar Threads
Was this page helpful?
