I have been facing this for past 5 days and hasn't been resolved yet, hence putting here:
I just crawled a site and waited after getting completed status for long time over 1-2 min and still the data isn't right. Now instead of sending just null in data, it is sending a list of nulls. Now this even breaks my checks, over if I received an empty list or a null value in data field.
The job_id is
56834938-0b41-458c-9012-9c7bcd7f7cbf
56834938-0b41-458c-9012-9c7bcd7f7cbf
Further, can't I access older crawls done using their job_ids? I get job_id doesn't for older jobs whereas they are there in the dashboard
The same is with every crawl I try can check job:
3232284b-94d1-4c95-90b9-d19065587d5b
3232284b-94d1-4c95-90b9-d19065587d5b
as well.
the data returned in API is of format:
"data": [null, null, null,....]
"data": [null, null, null,....]
As suggested I have delayed my final request to use the data by another 10 sec still the problem persists. Even trying to download the data from activity logs get's me this null data only.
Join builders, developers, users and turn any website into LLM-ready data, enabling developers to power their AI applications with clean, structured information crawled from the web.