Scraping process taking longer than expected for execution

Hi team,
Is someone else also experiencing slowness in the scrape jobs?
I'm seeing the scrape process is not able to scrape 200 odd pages in 15 minutes of time span.

Is this related to firecrawl servers or can it be related to the size of the webpages which I'm trying to scrape?
@Adobe.Flash @rafaelmiller

Caleb•8/14/24, 4:36 PM

Hey @Sachin, it could be related to both. What webpages are you trying to scrape?

SachinOP•8/15/24, 11:29 AM

Hi @Caleb , I was trying to scrape certain webpages for lifebit.ai in an iterative manner.

Also, I was facing the Rate Limit [429] error again yesterday during the scraping process. Could you please check again if things have changed for our account again?
TIA.

SachinOP•8/16/24, 12:01 PM

@Adobe.Flash @rafaelmiller for visibility.

Adobe.Flash•8/16/24, 2:55 PM

Hey @Sachin , no changes in your account. Everything seems to be good. It may have had that you hit the actual limit somehow, but on another note, we pushed our new architecture yesterday afternoon that will help with reducing unexpected errors

Adobe.Flash•8/16/24, 2:56 PM

ccing @rafaelmiller here to confirm via logs

SachinOP•8/16/24, 3:45 PM

Ok, thanks for the update.
Though I'm certain that I was well below my threshold for scraping requests per minute.

Will look into further.

Scraping process taking longer than expected for execution

Similar Threads

Similar Threads

Similar Threads