Fail to crawl paper information from the OnePetro website，return statuscode 403
Actually, when I use firecrawl, I try to crawl the paper information on the onepetro website (such as abstracts, etc.), but I found that the return status code 403 is returned when using the self-hosted mode, but it works normally when using the API. The curl code is the same, what could be the reason for this?
Curl: curl -X POST https://api.firecrawl.dev/v1/scrape \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer YOUR_API_KEY' \
-d '{
"url": "https://onepetro.org/search-results?page=1&q=knowledge%20graph",
"formats": [ "markdown" ]
}'
0 Replies