API Issue: Asynchronous Endpoint POST /v1/batch/scrape Fails, Synchronous Works
I have done extensive debugging and have isolated a critical issue with your API that is specific to my account.
Summary of Findings:
My API key is valid. The synchronous await app.batch_scrape_urls() method works correctly and processes URLs.
The asynchronous await app.async_batch_scrape_urls() method, which calls your POST /v1/batch/scrape endpoint, fails every time.
Instead of returning the expected JSON response ({"success": true, "id": "..."}), your API is returning a response that causes the Python SDK to resolve to None. This prevents me from getting a job_id and makes it impossible to use the asynchronous functionality, which is essential for my large-scale jobs.
Error:
Extraction pipeline complete! Successfully processed 0 URLs in this run.
Overall log now contains 58 successful records.
Overall 1 URLs failed across pre-flight and Firecrawl processing.
/usr/local/lib/python3.11/dist-packages/firecrawl/firecrawl.py:3369: RuntimeWarning: coroutine 'AsyncFirecrawlApp._handle_error' was never awaited
self._handle_error(response, 'start batch scrape job')
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
Traceback (most recent call last):
File "/tmp/ipython-input-10-199077524.py", line 191, in recover_or_start_batch_job
raise RuntimeError("Firecrawl API returned None when trying to start a batch job. Please check your API key and account status.")
RuntimeError: Firecrawl API returned None when trying to start a batch job. Please check your API key and account status.
Summary of Findings:
My API key is valid. The synchronous await app.batch_scrape_urls() method works correctly and processes URLs.
The asynchronous await app.async_batch_scrape_urls() method, which calls your POST /v1/batch/scrape endpoint, fails every time.
Instead of returning the expected JSON response ({"success": true, "id": "..."}), your API is returning a response that causes the Python SDK to resolve to None. This prevents me from getting a job_id and makes it impossible to use the asynchronous functionality, which is essential for my large-scale jobs.
Error:
Extraction pipeline complete! Successfully processed 0 URLs in this run.Overall log now contains 58 successful records.
Overall 1 URLs failed across pre-flight and Firecrawl processing.
/usr/local/lib/python3.11/dist-packages/firecrawl/firecrawl.py:3369: RuntimeWarning: coroutine 'AsyncFirecrawlApp._handle_error' was never awaited
self._handle_error(response, 'start batch scrape job')
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
Traceback (most recent call last):
File "/tmp/ipython-input-10-199077524.py", line 191, in recover_or_start_batch_job
raise RuntimeError("Firecrawl API returned None when trying to start a batch job. Please check your API key and account status.")
RuntimeError: Firecrawl API returned None when trying to start a batch job. Please check your API key and account status.
