CA
Crawlee & Apify•3y ago
NeoNomade

Setting cookies is failing

Error: 
/node_modules/puppeteer-core/lib/cjs/puppeteer/common/Connection.js:330
                error: new Errors_js_1.ProtocolError(),
                       ^

ProtocolError: Protocol error (Network.deleteCookies): Invalid parameters Failed to deserialize params.name - BINDINGS: mandatory field missing at position 6418
/node_modules/puppeteer-core/lib/cjs/puppeteer/common/Connection.js:330
                error: new Errors_js_1.ProtocolError(),
                       ^

ProtocolError: Protocol error (Network.deleteCookies): Invalid parameters Failed to deserialize params.name - BINDINGS: mandatory field missing at position 6418
Code : 
const goodCookies = session.getCookies('https://www.mywebsite.com/')
console.log(goodCookies) // I see the cookies and they look fine, a list of dicts.
page.setCookie(goodCookies)
const goodCookies = session.getCookies('https://www.mywebsite.com/')
console.log(goodCookies) // I see the cookies and they look fine, a list of dicts.
page.setCookie(goodCookies)
10 Replies
MEE6
MEE6•3y ago
@NeoNomade just advanced to level 5! Thanks for your contributions! 🎉
NeoNomade
NeoNomadeOP•3y ago
@Pepa J regarding cookies.
Pepa J
Pepa J•3y ago
@NeoNomade Are you using pupeteer? https://pptr.dev/api/puppeteer.page.setcookie then each of the cookies has to be provided as separate paramter. Try: 
await page.setCookie(...goodCookies)
await page.setCookie(...goodCookies)
Page.setCookie() method | Puppeteer
Signature:
NeoNomade
NeoNomadeOP•3y ago
Ok, then it also need to be changed in Crawlee docs, will try and come back with an answer
Pepa J
Pepa J•3y ago
may you provide the link for crawlee docs?
NeoNomade
NeoNomadeOP•3y ago
https://crawlee.dev/api/core/class/Session#getCookies
Session | API | Crawlee
Sessions are used to store information such as cookies and can be used for generating fingerprints and proxy sessions. You can imagine each session as a specific user, with its own cookies, IP (via proxy) and potentially a unique browser fingerprint. Session internal state can be enriched with custom user data for example some authorization toke...
NeoNomade
NeoNomadeOP•3y ago
same error. What I'm trying to achieve : do one request, get cookies from there, and use them in all the future requests.
Pepa J
Pepa J•3y ago
I am not sure do you mean by need to be changed in Crawlee docs . The getCookies is working fine, I was mentioning your usage of setCookies that it require to provide each cookie as separate parameter, which is same approach as in Puppeteer. I just did PoC: 
router.addHandler('detail', async ({ page }) => {
    /// ...
    const cookies = await page.cookies();

    await Actor.setValue('cookies', cookies);
    /// ...
});
router.addHandler('detail', async ({ page }) => {
    /// ...
    const cookies = await page.cookies();

    await Actor.setValue('cookies', cookies);
    /// ...
});
const crawler = new PuppeteerCrawler({
    proxyConfiguration,
    requestHandler: router,
    preNavigationHooks: [
        async ({page}) => {
            const storedCookies = await Actor.getValue("cookies");
            if (storedCookies) {
                await page.setCookie(...storedCookies);
            }
        }
    ]
});
const crawler = new PuppeteerCrawler({
    proxyConfiguration,
    requestHandler: router,
    preNavigationHooks: [
        async ({page}) => {
            const storedCookies = await Actor.getValue("cookies");
            if (storedCookies) {
                await page.setCookie(...storedCookies);
            }
        }
    ]
});
And it seems to be working fine. Once the cookies are stored, they are used in all upcoming requests.
NeoNomade
NeoNomadeOP•3y ago
The PoC looks nice, but how do I filter the cookies in the preNavigationHooks, just for one URL ? I have : url1 , url2 ... url 50. I want to take cookies from url1 and use them in url2...url50 still not working, I've pasted the entire cookies and error in paste bin https://pastebin.com/98kunwBf @Pepa J
Pepa J
Pepa J•3y ago
// main.js
import { Actor } from 'apify';
import { PuppeteerCrawler } from 'crawlee';
import { router } from './routes.js';

await Actor.init();

const startUrls = ['https://example.com']; // Inital URL where the cookies are taken from

const proxyConfiguration = await Actor.createProxyConfiguration();

const crawler = new PuppeteerCrawler({
    proxyConfiguration,
    requestHandler: router,
    maxConcurrency: 1, // make sure we start with only one request
    preNavigationHooks: [
        async ({page}) => {
            const storedCookies = await Actor.getValue("cookies");
            if (storedCookies) {
                await page.setCookie(...storedCookies);
            }
        }
    ]
});

await crawler.run(startUrls);

// Exit successfully
await Actor.exit();
// main.js
import { Actor } from 'apify';
import { PuppeteerCrawler } from 'crawlee';
import { router } from './routes.js';

await Actor.init();

const startUrls = ['https://example.com']; // Inital URL where the cookies are taken from

const proxyConfiguration = await Actor.createProxyConfiguration();

const crawler = new PuppeteerCrawler({
    proxyConfiguration,
    requestHandler: router,
    maxConcurrency: 1, // make sure we start with only one request
    preNavigationHooks: [
        async ({page}) => {
            const storedCookies = await Actor.getValue("cookies");
            if (storedCookies) {
                await page.setCookie(...storedCookies);
            }
        }
    ]
});

await crawler.run(startUrls);

// Exit successfully
await Actor.exit();
// routes.js
import { createPuppeteerRouter, utils } from 'crawlee';

export const router = createPuppeteerRouter();

router.addDefaultHandler(async ({ page, log, crawler }) => {
    crawler.autoscaledPool.maxConcurrency = 4;

    // Queue next requests, due to prenavigation hook they will be provided with the cookies
    await crawler.addRequests([
        { url: 'https://example.com/1', label: "OTHER" },
        { url: 'https://example.com/2', label: "OTHER"  },
        { url: 'https://example.com/3', label: "OTHER"  },
        { url: 'https://example.com/4', label: "OTHER"  },
        { url: 'https://example.com/5', label: "OTHER"  },
    ]);
});

router.addHandler('OTHER', async ({ page, log, crawler }) => {
    console.log(`My cookies are`, await page.cookies());
});
// routes.js
import { createPuppeteerRouter, utils } from 'crawlee';

export const router = createPuppeteerRouter();

router.addDefaultHandler(async ({ page, log, crawler }) => {
    crawler.autoscaledPool.maxConcurrency = 4;

    // Queue next requests, due to prenavigation hook they will be provided with the cookies
    await crawler.addRequests([
        { url: 'https://example.com/1', label: "OTHER" },
        { url: 'https://example.com/2', label: "OTHER"  },
        { url: 'https://example.com/3', label: "OTHER"  },
        { url: 'https://example.com/4', label: "OTHER"  },
        { url: 'https://example.com/5', label: "OTHER"  },
    ]);
});

router.addHandler('OTHER', async ({ page, log, crawler }) => {
    console.log(`My cookies are`, await page.cookies());
});
Sadly the link doesn't work for me 😦
how do I filter the cookies in the preNavigationHooks, just for one URL
You can set specific label for the request and then handle it this way in RouteHandler, you may also not use label and make this happen inside the defaultRouteHandler and then handle other requests in with routeHandler related to another label. I am not really sure about your use-case but I think that from this example you may get the idea.
I want to take cookies from url1 and use them in url2...url50
That is standard browser behavior if the urls share domain and there are no further restrictions on the cookies. In my propose when you do the first run the value storedCookies is not set. For the other requests 2-infinity the cookies were already stored in KV store, so they will be loaded and used from it.

Did you find this page helpful?