Effect CommunityEC
Effect Community•9mo ago
Ante D

Processing Large Arrays in Chunks with Effect Typescript

Hello everyone,

I have a very large array which I need to process and I want to do it in most effectful and optimal way.

My idea was to process it in chunks of 10 as each item in this array represents an ID.

Now for each of this ids we first need to query db to first get the new list of items and then update this items. 

const arrayInChunks = Chunk.fromIterable(veryLargeArray).pipe(Chunk.chunksOf(10))

  const updateTasks = Effect.forEach(arrayInChunks, (chunk) =>
    Effect.gen(function* () {
      const querySnapshot = yield* Effect.tryPromise({
        try: () => db.collection("tasks").where("taskId", "in", chunk).get(),
        catch: (error) => console.error(error),
      });

      const updateSubtasks = querySnapshot.docs.map((doc) =>
        Effect.tryPromise({
          try: () => db.collection("subtasks").doc(doc.id).update({ data }),
          catch: (error) => console.error(error),
        })
      );
      yield* Effect.all(updateSubtasks, { concurrency: "inherit" });
    })
  );


This is what I came up with but firstly I am not sure if this is best approach for doing this and secondly if this is good is there a way of making it even better
Was this page helpful?