Scraping websites with an added click needed
Hi guys i have recently been getting into ai / coding / automation through lots of different sites (Zapier, relevance, ect) now i have been given a task, it seems so damn easy but i have spent 2 days desperately trying to get this to work to no avail.
My task is I’m trying to scrape information from a website every Monday morning at 9am right but theres a catch, i cant just pull the web page i want because it returns an error, i think its because it needs to go to the search page first or something but the way i have to do it is go to url press the search button then scrape the search results, thats all! It seems so simple but yet i have just been struggling so bad.
I went through the fire crawl website and went to scrape then added the agent and put in the prompt “go to url, press search and scrape the search results” and IT WORKED!!! I was like f*** yes it can work but then sadly i discovered that when trying to do this in zapier so it can trigger at 9am every monday > go through a google spreadsheet 1 row at a time and get the urls > post them 1 by one to fire-crawl and then of course scrape the desired search data and add the results to a spreadsheet. It just returns the page that i don’t want the one before the click (using the scrape function i think I’m not to sure as I’m super new to all of this so apologies )
So i think i just need to find a way to do it on Zapier and somehow ask the agent to do that prompt through web hooks or something.
Chat gpt was not really any help which is usually my go to when im learning new things but im really stuck and need some help or ima be trialing and erroring for days if not weeks.
Please can anyone help me and if i haven’t made sense please just ask for clarification, really appreciate any response, thanks guys!
My task is I’m trying to scrape information from a website every Monday morning at 9am right but theres a catch, i cant just pull the web page i want because it returns an error, i think its because it needs to go to the search page first or something but the way i have to do it is go to url press the search button then scrape the search results, thats all! It seems so simple but yet i have just been struggling so bad.
I went through the fire crawl website and went to scrape then added the agent and put in the prompt “go to url, press search and scrape the search results” and IT WORKED!!! I was like f*** yes it can work but then sadly i discovered that when trying to do this in zapier so it can trigger at 9am every monday > go through a google spreadsheet 1 row at a time and get the urls > post them 1 by one to fire-crawl and then of course scrape the desired search data and add the results to a spreadsheet. It just returns the page that i don’t want the one before the click (using the scrape function i think I’m not to sure as I’m super new to all of this so apologies )
So i think i just need to find a way to do it on Zapier and somehow ask the agent to do that prompt through web hooks or something.
Chat gpt was not really any help which is usually my go to when im learning new things but im really stuck and need some help or ima be trialing and erroring for days if not weeks.
Please can anyone help me and if i haven’t made sense please just ask for clarification, really appreciate any response, thanks guys!
