n8n - looping through 300 records to search each result - 500 error
I am new to n8n + firecrawl. Building a workflow to iterate through 300 entries from json and firecrawl/search each. From n8n I am using generic HTTP POST with this payload:
{
"query": "{{ $json.name }}, {{$json.city}}, {{$json.state}}",
"sources": ["news"],
"location": "{{ $json.state }}",
"limit": 1,
"tbs": "qdr:w",
"scrapeOptions": {
"formats": ["summary","markdown"]
}
}
Items per batch: 1, Batch Interval 2000ms
Timeout 11000
I have it set to retry, but it still will kick a 500 error when iterating thru the loop. How can I fix this? My plan is a hobby plan, 50 requests/min ...I think my timing/logic is correct.
Please advise.
3 Replies
Hi @beardy does your Firecrawl API
/search works with static values for the above payload, if you test it on Postman or Hopscotch?
also, by default n8n nodes loop their input items.I set it in a loop/split into 30 per batch and added a wait 60s after the firecrawl call.
Appears to have fixed the issue.
The payload with variables was good-I was watching the logs on firecrawl side and data was clean.
Thanks for the tip on Postman/Hopscotch - I'll keep that in mind for testing in the future
Awesome, happy to help!