`crawl` results in `waiting` but `scrape` works
Hello, when running locally, I'm able to scrape, using curl, successfully.
However, if I try the crawl endpoint it results in a job that is constantly waiting.
Is this because it depends on scrapingbee?
I do see the following log which may be relevant:
The
However, if I try the crawl endpoint it results in a job that is constantly waiting.
Is this because it depends on scrapingbee?
I do see the following log which may be relevant:
Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-9.1.4.tgz
> [email protected] start:production /app
> tsc && node dist/src/index.js
Authentication is disabled. Supabase client will not be initialized.
POSTHOG_API_KEY is not provided - your events will not be logged. Using MockPostHog as a fallback. See posthog.ts for more.
Web scraper queue created
Server listening on port 3002
For the UI, open http://0.0.0.0:3002/admin//queues
1. Make sure Redis is running on port 6379 by default
2. If you want to run nango, make sure you do port forwarding in 3002 using ngrok http 3002
WARNING - You're bypassing authentication
WARNING - You're bypassing authentication
Attempted to access Supabase client when it's not configured.
Error logging crawl job:
Error: Supabase client is not configured.
at Proxy.<anonymous> (/app/dist/src/services/supabase.js:38:23)
at logCrawl (/app/dist/src/services/logging/crawl_log.js:9:14)
at crawlController (/app/dist/src/controllers/crawl.js:87:40)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
WARNING - You're bypassing authentication
WARNING - You're bypassing authentication
[Playwright] Error fetching url: https://www.stuff.co.nz/ with status: 404
Falling back to fetch
WARNING - You're bypassing authenticationCorepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-9.1.4.tgz
> [email protected] start:production /app
> tsc && node dist/src/index.js
Authentication is disabled. Supabase client will not be initialized.
POSTHOG_API_KEY is not provided - your events will not be logged. Using MockPostHog as a fallback. See posthog.ts for more.
Web scraper queue created
Server listening on port 3002
For the UI, open http://0.0.0.0:3002/admin//queues
1. Make sure Redis is running on port 6379 by default
2. If you want to run nango, make sure you do port forwarding in 3002 using ngrok http 3002
WARNING - You're bypassing authentication
WARNING - You're bypassing authentication
Attempted to access Supabase client when it's not configured.
Error logging crawl job:
Error: Supabase client is not configured.
at Proxy.<anonymous> (/app/dist/src/services/supabase.js:38:23)
at logCrawl (/app/dist/src/services/logging/crawl_log.js:9:14)
at crawlController (/app/dist/src/controllers/crawl.js:87:40)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
WARNING - You're bypassing authentication
WARNING - You're bypassing authentication
[Playwright] Error fetching url: https://www.stuff.co.nz/ with status: 404
Falling back to fetch
WARNING - You're bypassing authenticationThe
404404