Crawlee & Apify•2y ago

cookies help

Hello, I've done first the experiment in postman. Visit a page, take the cookies from there, go to another page ( that required those cookies ) and it worked. I'm trying to do the same thing with Cheerio, but it fails. Here is my code :

let popReportCookies;
router.addDefaultHandler(async ({ enqueueLinks, log, session, crawler, request }) => {
    log.info(request.loadedUrl)
    popReportCookies = session.getCookieString('https://www.example.com/grading/pop-report')
    for (const setId of setIds) {
      let setUrl = `https://www.example.com/grading/set_match/${setId}`
      const request = {
        url: setUrl,
        headers: {
          cookie: popReportCookies,
        },
    };
    await crawler.requestQueue.addRequest(request, {forefront: true});
  }
  await enqueueLinks({
    globs: ['https://www.example.com/grading/set_item_listing_grade?set_id=**'],
    label: 'setItemListing'
  });
});

let popReportCookies;
router.addDefaultHandler(async ({ enqueueLinks, log, session, crawler, request }) => {
    log.info(request.loadedUrl)
    popReportCookies = session.getCookieString('https://www.example.com/grading/pop-report')
    for (const setId of setIds) {
      let setUrl = `https://www.example.com/grading/set_match/${setId}`
      const request = {
        url: setUrl,
        headers: {
          cookie: popReportCookies,
        },
    };
    await crawler.requestQueue.addRequest(request, {forefront: true});
  }
  await enqueueLinks({
    globs: ['https://www.example.com/grading/set_item_listing_grade?set_id=**'],
    label: 'setItemListing'
  });
});

9 Replies

manual-pink•2y ago

Hey there! I did not try to run it. But from the top of my head - if you would use the browser - the cookies are saved to session after requests is finished successfully. So in case of a browser - you will normally got something like await page.cookies() to grab the cookies. In case of cheerio - I'd say you should be looking for some response header with somewhat like headers['set-cookie'] (not an exact naming).

NeoNomadeOP•2y ago

hmm ok @Andrey Bykov succesfully extracted them, but now , setting them as headers while I create a new request is fine ? or should I do something else ?

manual-pink•2y ago

Yeah, should be exactly as you say. Basically set-cookie header are cookies that normally will be assigned to page. Then you save them as cookie header for next requests, and it should be good to go

NeoNomadeOP•2y ago

const request = {
        url: setUrl,
        headers: testHeaders
        };
        await crawler.requestQueue.addRequest(request, {forefront: true});

const request = {
        url: setUrl,
        headers: testHeaders
        };
        await crawler.requestQueue.addRequest(request, {forefront: true});

but it doesn't overwrite the other headers ? it just updates them ? in the testHeaders var I only have the set-cookie header

manual-pink•2y ago

you mean the fingerprint headers? Yeah, I think it should combine them, or do you assign some other headers elsewher?

NeoNomadeOP•2y ago

no just this place I want to be sure that they are combined :)) and I'm not sending just some cookies

manual-pink•2y ago

as I mentioned - it should be the case. Not sure though if the header should be 'set-cookie' or just 'cookie' - but you could try both I guess

NeoNomadeOP•2y ago

It should or it shouldn’t ? :))

WARN  CheerioCrawler: Reclaiming failed request back to the list or queue. Header name must be a valid HTTP token [:status]

WARN  CheerioCrawler: Reclaiming failed request back to the list or queue. Header name must be a valid HTTP token [:status]

manual-pink•2y ago

:status is not a valid header indeed. Could you provide the repro or something? It should be that with request you get a response with headers (cookie, set-cookie, something like that). Then you take the value and use it as cookie header for next requests...

Gaming

Programming

cookies help

Did you find this page helpful?