Edge Functions, Massive Concurrencies & Failed Function Invocations
So let me explain this scenario and provide some insight so that it's easy to demonstrate what's happening here and hopefully it's enough insight to explain what's happening here. I'm unsure if I'm just missing something or my implementation is faulty, but any help is much appreciated.
I have a process for ingesting PDFs to create embeddings for customers, such that for every chunk of text from that PDF, a new Edge Function is spawned to ingest that chunk (some PDFs have thousands of chunks). This creates a scenario where you have large amounts of parallelism and lots of concurrent Edge Functions being invoked for the same process.
The problem is that when ingesting one file, there is no problem (except for large files, it will stop after about 500-700 chunks), but when I ingest multiple files at the same time, it goes absolutely haywire. I've observed some files completing entirely, some files getting about 30-40% through and then suddenly stopping, some just outright failing to even ingest one chunk.
The question I have is, are there rate limits for Supabase Edge Functions, such that my massive requests for 1000-1500 Edge Function invocations in the span of milliseconds is too much? Or is there some sort of thread pooler that can only allow a maximum number of active functions at any given time? Much appreciated, I'll submit a support request if no one here has any insight, but hopefully that's not the case
I have a process for ingesting PDFs to create embeddings for customers, such that for every chunk of text from that PDF, a new Edge Function is spawned to ingest that chunk (some PDFs have thousands of chunks). This creates a scenario where you have large amounts of parallelism and lots of concurrent Edge Functions being invoked for the same process.
The problem is that when ingesting one file, there is no problem (except for large files, it will stop after about 500-700 chunks), but when I ingest multiple files at the same time, it goes absolutely haywire. I've observed some files completing entirely, some files getting about 30-40% through and then suddenly stopping, some just outright failing to even ingest one chunk.
The question I have is, are there rate limits for Supabase Edge Functions, such that my massive requests for 1000-1500 Edge Function invocations in the span of milliseconds is too much? Or is there some sort of thread pooler that can only allow a maximum number of active functions at any given time? Much appreciated, I'll submit a support request if no one here has any insight, but hopefully that's not the case
