CA
Crawlee & Apify•2mo ago
unwilling-turquoise

Request works in Postman but doesn't works in crawler even with full browser

Hello I'm trying to handle ajax call via got-scraping. I prepare call in postman, where it works fine. But if I want to try it in Actor a got 403 every time. Even if I try i via Puppeteer or Playwrite and click on the button with request I got response with geo.captcha-delivery.com/captcha url to solve it. Please can anybody give me any advice how to handle this issue?
6 Replies
Hall
Hall•2mo ago
Someone will reply to you shortly. In the meantime, this might help: -# This post was marked as solved by Matous. View answer.
stormy-gold
stormy-gold•2mo ago
Hi, did you try to fully recreate all the headers that Postman creates automatically for you? It might be that you need to add Content-Type or User-Agent for example. Another thing that could be causing the issue are the proxies, with Postman you are using your home IP, but if you create the actor you might be using different proxies, depending where you run it and if you do use the apify proxy configuration.
unwilling-turquoise
unwilling-turquoiseOP•2mo ago
But I used RESIDENTIAL proxies from particular country.
Louis Deconinck
Louis Deconinck•5w ago
Perhaps you can share some specific code and URLs. Right now it's just guessing what might be going wrong.
unwilling-turquoise
unwilling-turquoiseOP•5w ago
yes of course I have this website, for example this detailed URL: https://www.idealista.com/inmueble/98545124/ On this detail there is a link to view the mobile phone downloaded using the ajax call "Ver teléfono". https://www.idealista.com/inmueble/98545124/ I am able to scrape this detail but when try to call phone url .. every time got 403 ... even if I start browser, click on that link and catch response... I got 403 status code I tried to use gotScraping 
 const gotResponse = await gotScraping({
        url: `https://www.idealista.com/${pageCountry}/ajax/ads/${output.id}/contact-phones`,
        headers:{
            accept:'application/json, text/javascript, */*; q=0.01',
            'accept-language':'en-US,en;q=0.9',
            referer: output.url,
            'user-agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36',
            'x-requested-with':'XMLHttpRequest',
            Connection: 'keep-alive',
            // Cookie: cookieHeader,
        },
        proxyUrl: await proxyConfiguration.newUrl(session.id),
    });

    console.log(gotResponse.statusCode);
 const gotResponse = await gotScraping({
        url: `https://www.idealista.com/${pageCountry}/ajax/ads/${output.id}/contact-phones`,
        headers:{
            accept:'application/json, text/javascript, */*; q=0.01',
            'accept-language':'en-US,en;q=0.9',
            referer: output.url,
            'user-agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36',
            'x-requested-with':'XMLHttpRequest',
            Connection: 'keep-alive',
            // Cookie: cookieHeader,
        },
        proxyUrl: await proxyConfiguration.newUrl(session.id),
    });

    console.log(gotResponse.statusCode);
but the result is same
other-emerald
other-emerald•4w ago
It may be that you are missing some cookies. On the request generated for the phone number I can see there is. SESSION and a userUUID cookie so maybe you have to get those. If you are using Playwright or Puppeteer you can also use evaluate to perform the request in the browser console https://playwright.dev/docs/evaluating https://pptr.dev/api/puppeteer.page.evaluate

Did you find this page helpful?