Crawlee & Apify•4mo ago

How can I add my own cookies to the crawler

Hi, I'm using crawlee to fetch some data but I don't know how to add my own cookies in my crawler. I'm using Playwright to fetch cookies and after that I want to pass (in a session if it is possible) them to my BeautifulSoupCrawler.

4 Replies

Hall•4mo ago

Someone will reply to you shortly. In the meantime, this might help:

xenophobic-harlequin•4mo ago

Hey You can pass cookies to Request headers. If you want to deal with cookies at the session level, now this approach varies depending on the client you are using. See - https://github.com/apify/crawlee-python/issues/933

extended-salmonOP•4mo ago

Thanks @Mantisus . This was my soluction and it works for me:

        cookie_header = "; ".join([f"{cookie['name']}={cookie['value']}" for cookie in cookies])
        # type of cookies is a list of dicts
        headers = {
            'content-type': 'application/x-www-form-urlencoded',
            'accept-language': 'en-US,en;q=0.9',
            'cookie': cookie_header,
        }

        initial_req = Request.from_url(
                url=url,
                method="POST",
                label="LISTINGS",
                headers=headers,
                payload=urlencode(get_payload(0)).encode(),
                use_extended_unique_key=True,
                user_data={
                    "page_index": 0,
                    "headers": headers  
                },
            )
        
        await crawler.run([initial_req])

        cookie_header = "; ".join([f"{cookie['name']}={cookie['value']}" for cookie in cookies])
        # type of cookies is a list of dicts
        headers = {
            'content-type': 'application/x-www-form-urlencoded',
            'accept-language': 'en-US,en;q=0.9',
            'cookie': cookie_header,
        }

        initial_req = Request.from_url(
                url=url,
                method="POST",
                label="LISTINGS",
                headers=headers,
                payload=urlencode(get_payload(0)).encode(),
                use_extended_unique_key=True,
                user_data={
                    "page_index": 0,
                    "headers": headers  
                },
            )
        
        await crawler.run([initial_req])

I use the headers in the next request (in the handler) that I have passed in user_data

MEE6•4mo ago

@frankman just advanced to level 2! Thanks for your contributions! 🎉

Gaming

Programming

How can I add my own cookies to the crawler

Did you find this page helpful?