CA
foreign-sapphire
How do you add custom link 🔗
Hello 😊
In most cases is use the enqueue_links to add links via a selector. On rare occasions, I need to add a custom link with a label when crawling (adding a calulcated ?page=x to the url). How can I do that?
5 Replies
you can use this in your handler:
await crawler.addRequests([...]);foreign-sapphireOP•11mo ago
Thanks! I did not know about that. How do I add a label(s) when using addRequests? It does not seem to have such parameter or args.
like this >
await crawler.addRequests([{url:'some url', label: 'some label', userData: {....}}, {another request object}, ... ]);foreign-sapphireOP•11mo ago
Thank you so much!
Just for reference. The dict in a list threw an error. If you want to create a request you can do it like this:
from crawlee.models import BaseRequestData
from crawlee._utils.requests import compute_unique_key
total_products_url = context.request.url + '?page=' + str(int(number_of_pages))
new_request = BaseRequestData(unique_key=compute_unique_key(total_products_url), url=total_products_url, user_data= {'label': 'ALL_PRODUCTS_IN_CATEGORY'})
await crawler.add_requests([new_request])Oh, sorry I did not noticed that this is Python question. That example was for javascript but I am glad that you made it work. 👍