/scrape Endpoint fails with 500, can i see the error message somewhere in the dashboard?
Like the title says. I fired a call to the /scrape endpoint with an url and get a 500 error. Am i allowed to post the job id and or the url to the page?
15 Replies
Hey @endless90 can you share which URL you are having trouble with? I can see if I can reproduce it.
iam trying to scrape the "unsere wochenkarte" section of this page https://www.cafebar-central.de/speisekarte
central
Wochenkarte/Speisekarte | central
Speisekarte - Tageskarte - Wochenkarte
"regional• bio• mediterran • saisonal • vegan•Frühstück•Mittagstisch•hausgemachte Kuchen•welchselnde Tageskarte• erlesene Weine• glutenfrei• Gutscheine • Kaffeespezialitäten• täglich geöffnet • Nachhaltigkeit• wir kochen frisch ..... und vieles mehr"
the api. ill later paste the prompt and settings here. thanks for the help
Sounds good! Once you share, I can try to reproduce it locally.
this is the request with actions. sadly this returns 500 because it cant find the element. maybe you have a hint for me why it cant be found.
the other request withour actions now just times out. maybe it is a problem with charles proxy. ill disable the proxy and try again
yeah that worked. but sadly without actions so i cannot click on the second tab on the page.
this is interesting. when i use the playground ill get the "wochengerichte" without any actions or any settings. you can see that because it got the "Asia Nudelsuppe" which is on the second page.
when i run my code and send the prompt over the api i only get the first page. maybe i have to add one actions which waits for 2 seconds.
hmm sadly that did not work
@endless90 to debug why it's not finding the link, I would try two things:
- use the "screenshot" action so that you can see what Firecrawl sees: https://docs.firecrawl.dev/advanced-scraping-guide#screenshot
- ask for the "rawHtml" of the page, so that you can confirm that the HTML entity that you are looking for is actually there
Another idea is to trying a simpler selector:
#menu0c689dce-b142-432f-bcdd-ddb0e29edfc1. Seems brittle since I'm not sure what will cause the UUID in there to change, but depending on what your use case is, this may be sufficient.damn seems like i get a timeout now. i added rawHtml, html and the screenshot action.
thats the job id c8047128-da10-4d4b-81ae-4d88f7b3fd95
maybe raw html and html are too much for the crawler
I tested a couple of requests. And i think it has to do something with the wix page. When i dont wait for it to load it sometimes work. When i add a wait action > 1 second it crashes with an 500 error. if i use my private page it works like a charm
hey @micah.stairs i saw firecrawl had a couple of updates. now i tried it again but sadly i still get an timeout error when i try to scrape the WIX iframe page.
I attached my request
