Crawlee & Apify•2mo ago

TypeError: Invalid URL

Adding requests with crawler.run(["https://website.com/1234"]); works locally while in the apify cloud it breaks with the following error: Reclaiming failed request back to the list or queue. TypeError: Invalid URL It appears that while running in the cloud, the URL is split by character and each creates a request in the queue, as it can be seen in the screenshot. The bug happens no matter the URL is hardcoded in the code or added dynamically via input. I'm using crawlee 3.13.0. Complete error stack:

WARN  PlaywrightCrawler: Reclaiming failed request back to the list or queue. TypeError: Invalid URL
2025-03-11T19:21:27.987Z     at new URL (node:internal/url:806:29)
2025-03-11T19:21:27.988Z     at getCookieContext (/home/myuser/node_modules/tough-cookie/dist/cookie/cookieJar.js:75:20)
2025-03-11T19:21:27.989Z     at CookieJar.getCookies (/home/myuser/node_modules/tough-cookie/dist/cookie/cookieJar.js:452:23)
2025-03-11T19:21:27.989Z     at CookieJar.callSync (/home/myuser/node_modules/tough-cookie/dist/cookie/cookieJar.js:168:16)
2025-03-11T19:21:27.990Z     at CookieJar.getCookiesSync (/home/myuser/node_modules/tough-cookie/dist/cookie/cookieJar.js:575:21)
2025-03-11T19:21:27.991Z     at Session.getCookies (/home/myuser/node_modules/@crawlee/core/session_pool/session.js:264:40)
2025-03-11T19:21:27.992Z     at PlaywrightCrawler._applyCookies (/home/myuser/node_modules/@crawlee/browser/internals/browser-crawler.js:344:40)
2025-03-11T19:21:27.992Z     at PlaywrightCrawler._handleNavigation (/home/myuser/node_modules/@crawlee/browser/internals/browser-crawler.js:329:20)
2025-03-11T19:21:27.993Z     at async PlaywrightCrawler._runRequestHandler (/home/myuser/node_modules/@crawlee/browser/internals/browser-crawler.js:260:13)
2025-03-11T19:21:27.994Z     at async PlaywrightCrawler._runRequestHandler (/home/myuser/node_modules/@crawlee/playwright/internals/playwright-crawler.js:114:9) {"id":"PznVw0jlt50G6EL","url":"D","retryCount":1}

WARN  PlaywrightCrawler: Reclaiming failed request back to the list or queue. TypeError: Invalid URL
2025-03-11T19:21:27.987Z     at new URL (node:internal/url:806:29)
2025-03-11T19:21:27.988Z     at getCookieContext (/home/myuser/node_modules/tough-cookie/dist/cookie/cookieJar.js:75:20)
2025-03-11T19:21:27.989Z     at CookieJar.getCookies (/home/myuser/node_modules/tough-cookie/dist/cookie/cookieJar.js:452:23)
2025-03-11T19:21:27.989Z     at CookieJar.callSync (/home/myuser/node_modules/tough-cookie/dist/cookie/cookieJar.js:168:16)
2025-03-11T19:21:27.990Z     at CookieJar.getCookiesSync (/home/myuser/node_modules/tough-cookie/dist/cookie/cookieJar.js:575:21)
2025-03-11T19:21:27.991Z     at Session.getCookies (/home/myuser/node_modules/@crawlee/core/session_pool/session.js:264:40)
2025-03-11T19:21:27.992Z     at PlaywrightCrawler._applyCookies (/home/myuser/node_modules/@crawlee/browser/internals/browser-crawler.js:344:40)
2025-03-11T19:21:27.992Z     at PlaywrightCrawler._handleNavigation (/home/myuser/node_modules/@crawlee/browser/internals/browser-crawler.js:329:20)
2025-03-11T19:21:27.993Z     at async PlaywrightCrawler._runRequestHandler (/home/myuser/node_modules/@crawlee/browser/internals/browser-crawler.js:260:13)
2025-03-11T19:21:27.994Z     at async PlaywrightCrawler._runRequestHandler (/home/myuser/node_modules/@crawlee/playwright/internals/playwright-crawler.js:114:9) {"id":"PznVw0jlt50G6EL","url":"D","retryCount":1}