FirecrawlF
Firecrawl•2y ago
Magick

`crawl` results in `waiting` but `scrape` works

Hello, when running locally, I'm able to scrape, using curl, successfully.

However, if I try the crawl endpoint it results in a job that is constantly waiting.

Is this because it depends on scrapingbee?

I do see the following log which may be relevant:

Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-9.1.4.tgz

> firecrawl-scraper-js@1.0.0 start:production /app
> tsc && node dist/src/index.js

Authentication is disabled. Supabase client will not be initialized.
POSTHOG_API_KEY is not provided - your events will not be logged. Using MockPostHog as a fallback. See posthog.ts for more.
Web scraper queue created
Server listening on port 3002
For the UI, open http://0.0.0.0:3002/admin//queues

1. Make sure Redis is running on port 6379 by default
2. If you want to run nango, make sure you do port forwarding in 3002 using ngrok http 3002 
WARNING - You're bypassing authentication
WARNING - You're bypassing authentication
Attempted to access Supabase client when it's not configured.
Error logging crawl job:
 Error: Supabase client is not configured.
    at Proxy.<anonymous> (/app/dist/src/services/supabase.js:38:23)
    at logCrawl (/app/dist/src/services/logging/crawl_log.js:9:14)
    at crawlController (/app/dist/src/controllers/crawl.js:87:40)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
WARNING - You're bypassing authentication
WARNING - You're bypassing authentication
[Playwright] Error fetching url: https://www.stuff.co.nz/ with status: 404
Falling back to fetch
WARNING - You're bypassing authentication


The 404 status seems misleading as the same url works from the scrape endpoint.
Was this page helpful?