Effect CommunityEC
Effect Community•2y ago•
11 replies
Néstor

Concurrent fetch calls

Hello ! Very new to effect. I'm trying to do a little script. The idea is as follows;

1. Grab a sitemap.xml from a fetch request.
2. Parse said sitemap.xml and perform one req per each of its items.
3. Try to make up to 10 concurrent fetch request. But no more.

This is the implementation I've done
const fetchSitemap = () =>
  Effect.tryPromise(() =>
    client.get(url, params, { headers }).then((response) => response.data)
  );

const parseSitemap = (data: string) =>
  Effect.tryPromise(() => parseStringPromise(data));

const fetchUrl = (url: string) =>
  Effect.tryPromise(() => {
    console.log(`Fetching URL: ${url}`);
    return client.get(url, params, { headers });
  });

const processUrlsWithConcurrency = (urls: string[], concurrencyLimit: number) =>
  Effect.gen(function* () {
    yield* Effect.all(
      urls.map(
        (u) =>
          fetchUrl(u).pipe(
            Effect.map(() => true),
            Effect.catchAll((error) =>
              Effect.logError(`Error fetching URL ${url}: ${error.message}`)
            )
          ),
        { concurrency: concurrencyLimit }
      )
    );
  });

const program = Effect.gen(function* () {
  // Fetch the sitemap
  yield* Effect.logInfo("Fetching sitemap...");
  const sitemapData = yield* fetchSitemap();

  // Parse the sitemap
  const parsedSitemap = yield* parseSitemap(sitemapData as string);
  yield* Effect.logInfo("Parsed sitemap");
  // Extract URLs
  const urls = parsedSitemap.urlset.url.map(
    (entry: any) => entry.loc[0] as string
  );

  // Fetch each URL with concurrency limit
  yield* processUrlsWithConcurrency(urls, 10);
});

Effect.runPromise(program).then(() => Effect.logInfo("All tasks completed"));


A few questions that arises are.
- How can I get ride of the console.log and use Effects logs, since I believe I have to yield them I need to make everything a generator ?
- I feel the Effect.all runs the Effects concurrently but not in parallel is this true ? I might be mixing topics.
- Any other improvement you feel is necessary is appreciated.
Was this page helpful?