How to send an URL with a label to main file?
I am trying to send an URL with a label and user data to main file in order to run this url directly from a specific handler within routes file. Is that possible? I am using Playwright.
5 Replies
Someone will reply to you shortly. In the meantime, this might help:
wee-brown•2w ago
Sorry, could you please give an example of the code. I don't quite understand your question
eastern-cyanOP•6d ago
Hi @Mantisus , thank you for your reply. I'm trying to send a specific URL from main.py file to the handler 'PRODUCT' in routes.py file with the label 'PRODUCT' and the 'id' as user_data. I'm doing that because depending on the URL I have to send it to one handler or another, and I need to send a custom label and user_data from the main.py file.
This is the sample code:
main.py
async def main(url: str) -> None:
crawler = PlaywrightCrawler(
request_handler=router
)
routes.py
@router.default_handler
async def request_handler(context: PlaywrightCrawlingContext) -> None:
# do something
await context.enqueue_links(
selector='test',
label='PRODUCT',
user_data = {"id": id}
)
@router.handler('PRODUCT')
async def product_handler(context: PlaywrightCrawlingContext) -> None:
context.log.info(f'product_handler is processing {context.request.url}')
id = context.request.user_data["id"]
Thankswee-brown•6d ago
Thanks for your example, you can use
Request with the from_url constructor, for this:
eastern-cyanOP•4d ago
Thank you @Mantisus that works perfectly. You are my hero!!!