Using userData in the queue from the API
Not using JS or python so I have to interact with the Apify API directly.
I need to store arbitrary data in request items in the queue, I've seen I can use the
userData
field when posting a request, but when getting a request from the head (https://docs.apify.com/api/v2#/reference/request-queues/queue-head/get-head) the response does not contain this userData
field. Instead i get the request id from this response, and have to make a second API call to get the details for a specific request (https://docs.apify.com/api/v2#/reference/request-queues/request/get-request) based on the request ID.
That's 2 API calls for 1 item from the queue, is there a better way?
Why doesn't the Get head
endpoint return the complete request (including userData
)?
I probably missed something there, thanks for your help.3 Replies
genetic-orange•12mo ago
Hello @Jerome,
you can't do this in one request, but you can fetch multiple request IDs using the get head call, which might help to optimize your solution.
If you'd be interested, here is the request fetching logic from crawlee:
https://github.com/apify/crawlee/blob/master/packages/core/src/storages/request_queue_v2.ts#L100
GitHub
crawlee/packages/core/src/storages/request_queue_v2.ts at master · ...
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, an...
optimistic-goldOP•12mo ago
Hi @vojtechmaslan , thanks for your answer.
Yes I read the python version of that in the apify SDK.
I'm curious at to why the row is filtered to return only some values?
genetic-orange•11mo ago
I will try to get some info from the platform team.
Hello @Jerome, I apologize for the delayed answer. This is intentional because userData field can be very big in size. There is currently no plan to change this.