Unable to scrape indeed url

It was working fine till yesterday.

{
"url": "https://ca.indeed.com/viewjob?jk=89d7f1cc3054c126&tk=1ibcemdv6jn2a801&from=hp&vjs=3&advn=7919383835287656&adid=384927630&ad=-6NYlbfkN0BkvSqPB7txKGhOQSuBkqljSXIPNNOywyQc03G4_L-y5zqtSmULOhCauyfaSLqGHXS23a8CzPu7re4alPi6E8hMqw5s3cEHYxZjG8OGoxz_BF_IfxuAlHg56GYAzmMjceZuvoV8s44gl01LNpYCVEX0lfWxuYaDVo3pJssRZzQSxfWsFAi6s0OlPQNbJoJL1MEAw2Rix8gt6acdUakJm8Cb0N32fGTt9nkyGSaA3MyCbNpeFkh2XE89A_4O0WuK1Asr16YTRYtcAEd_yVqLyWtfDxuC39rPOBIO6sQ8AOV0jNp0tfQWykAG2WWxIhHrZAH4tP0t4k4BrEY5RaMDpA7OsFoPq19tvgZiMhFgdPAGAiVdgcQZhkViA9PNKav9b90hoBzfQUf2q_rZTSfXoQ_BkAEG284hVf4tq6zXXXYoI5mA0b-r30rjZHnYBt3D6Vr7wM8ENQ9QdeBwaQo3JpCJZRnPHr0UOTJJG5lVd4zWww7UpxAEhloOUNGxz2EKnbtzkDpu8wKctrz5_Y0iNN0qURMfG8FA7F8PFaedc9sdnp8H6QX8xr0so5AhDuGDWWJnM2zjh_Y0LDCbaSswRO3fQNw7FsQV0OXw5RdiImZ3Nw==&xkcb=SoD76_M36sVWvywB5J0LbzkdCdPP&xpse=SoA06_I36sVWZGR7n50IbzkdCdPP&sjdu=o4-SOnWFj7zDQa1x_oNfXdq7ED1XT5Bb9w9Crk2BBM1TaV54WyBRVunvknJ4haBtDHorEu9E3Ggx0ZUSwOza2kECv-r3eSkK-iwMwl6MMJSNRL2n3bCrle9sRAzZvag-2_iCukWB1z7cU3HtF2xwj_O6G10EUUKLg5TyLOIYngRXNZI9_2MVj3m9UlTtZSnr9M4di0luNfVKRxCQMLM8_l_jZMIX13PdEexXVjH7xBrh6ncAorSjBiJBgA5ceMaQJJIeA7u8IlESiGla7gJyNA",
"type": "scrape",
"method": "f-eng",
"result": {
"error": "Forbidden",
"success": false,
"time_taken": 554,
"response_code": 403,
"response_size": 62577
},
"createdAt": "2024-10-29T15:41:20.708109+00:00"
}

xasdasdasOP•10/29/24, 7:40 PM

When usign API:

{
"markdown":"Additional Verification Required\n================================\n\nVerifying you are human. This may take a few seconds.\n\nWaiting for ca.indeed.com to respond...\n\nPlease enable Cookies and reload the page.\n\nYour Ray ID for this request is 8da5a0a63b3b0850\n\nNeed more help? Contact us",
"html":"<!DOCTYPE html><html lang="en"><body dir="ltr"> <main class="error"> <h1>Additional Verification Required</h1> <p></p><div><p id="cf-spinner-please-wait" style="display: block; visibility: visible;">Verifying you are human. This may take a few seconds.</p><p id="cf-spinner-redirecting" style="display:none">Waiting for ca.indeed.com to respond...</p></div><div id="no-cookie-warning" style="display:none"><p>Please enable Cookies and reload the page.</p></div><form id="challenge-form"><div id="gLIfn4" style="display: none;"></div></form><p></p> <p>Your Ray ID for this request is <span style="font-style:italic">8da5a0a63b3b0850</span></p> <p>Need more help? <a href="https://www.indeed.com/support/contact\">Contact us</a></p> </main> </body></html>",
"metadata":{
"title":"Security Check - Indeed.com",
"language":"en",
"ogLocaleAlternate":[

],
"viewport":"width=device-width, initial-scale=1",
"sourceURL":"https://ca.indeed.com/viewjob?jk=cebc1cb267920fc1&tk=1ibcdb6ghj72s800&from=hp&xpse=SoCg67I36sxIjZXzQx0LbzkdCdPP&xfps=6acb753a-8ef9-43bb-94af-0eb6db47e6aa&xkcb=SoBU67M36sSjOFwByp0LbzkdCdPP",
"error":"Forbidden",
"statusCode":403
}

rafaelmiller•10/30/24, 12:25 PM

hey @xasdasdas this seems like a case of robot blocking behavior. I'm ccying our scraping engineer @thomas to investigate further. I'll close this thread, and we can continue the conversation via email.

Unable to scrape indeed url

Similar Threads

Similar Threads

Similar Threads