Need help: next page extraction
Hi
Unfortunately, I am not able to extract product information of the subsequent pages, just for the initial page. My assumption: all subsequent pages have the same URL? But who am I - I have no clue and no DEV experience 🙂 !
https://www.coop.ch/de/lebensmittel/milchprodukte-eier/c/m_0055?page=1&pageSize=60&q=%3Arelevance&sort=relevance
Can anyone help - would be highly appreciated! 😍
8 Replies
It doesn't look like the subsequent pages have the same URL. The page number parameter changes in the url. I don't believe extract function goes through multiple pages, only the page that is provided. You could look at what the min and max page numbers are and run extract on each page.
^ Correction, extract can run on multiple page, you have to add * as wildcard to the URL
Thanks @ayang. ! I found out, that after the first page only the URL changes and the number of the page withing the URL https://www.coop.ch/de/lebensmittel/milchprodukte-eier/c/m_0055?page=3&q=%3Arelevance&sort=relevance ...so page=4, page=5....so that does not work as the "*" would be in the middle of the URL...at least that is my "newbie" explanation 🙂
We can use for loop with the page numbers, edit the page url and run extract individually for each page inside the for loop
Thanks again @ayang. 👍 . what do you mean "for loop". Is this a way I could "automate" with firecrawl? I need to scrape around 800 pages...
@navy664 are you using the playground or api? for loop would be for using the api.
@ayang. again, thank you so much. I used playground! Ok I will try to set everything up with api (with the help of chatgtp or claude 😀 ). Last question: so I guess I can handle with the API also such cases as with this website (I get 0 extracted products with playground): https://www.migros.ch/de/category/fruchte-gemuse?page=3
Thanks again
Migros
Früchte & Gemüse • Migros
Frische Lebensmittel, Tiefkühlkost und alles für den Haushalt online bestellen. 12 500 Migros Produkte & Markenartikel zum Migros Preis.
yes I would think so. For this page, you might also consider using "actions" since it looks like its a single button pressed at the bottom of the page to "show more"
@ayang. Thank you soo much!👍