Running locally: "The engine used does not support the following features: waitFor"
I run the latest version from Github on localhost. I can't use the v1 scrape with "waitFor". I get the warning:
The engine used does not support the following features: waitFor -- your scrape may be partial.The log says "scraping via fetch". Which engines are at play here and how to use them? I can't find a parameter for that.
2 Replies
You need to set up the playwright microservice
@mogery that was a great hint - thank you. I have it running now at http://localhost:3000/scrape - I can successfully call it from postman and it scrapes a page. However, it's not called by firecrawl although it's in .env.local
NUM_WORKERS_PER_QUEUE=8
PORT=3002
HOST=0.0.0.0
REDIS_URL=redis://localhost:6379 #for self-hosting using docker, use redis://redis:6379. For running locally, use redis://localhost:6379
REDIS_RATE_LIMIT_URL=redis://localhost:6379 #for self-hosting using docker, use redis://redis:6379. For running locally, use redis://localhost:6379
PLAYWRIGHT_MICROSERVICE_URL=http://localhost:3000/scrape
## To turn on DB authentication, you need to set up supabase.
USE_DB_AUTHENTICATION=false
Any idea what I do wrong?
Log of the console:
gunter@W-PF4NB8HE:~/firecrawl/firecrawl/apps/api$ pnpm run workers firecrawl-scraper-js@1.0.0 workers /home/gunter/firecrawl/firecrawl/apps/api nodemon --exec ts-node src/services/queue-worker.ts...
2024-11-18 20:33:14 debug [ScrapeURL:]: Engine fetch meets feature priority threshold 2024-11-18 20:33:14 debug [ScrapeURL:]: Engine pdf meets feature priority threshold 2024-11-18 20:33:14 debug [ScrapeURL:]: Engine docx meets feature priority threshold 2024-11-18 20:33:14 info [ScrapeURL:]: Scraping via fetch... 2024-11-18 20:33:14 info [ScrapeURL:]: Scrape via fetch deemed successful. 2024-11-18 20:33:14 warn [ScrapeURL:]: The engine used does not support the following features: waitFor -- your scrape may be partial. {"module":"ScrapeURL","scrapeId":"b29473ef-d6ce-4683-93cb-78a21fe1822c","scrapeURL":"https://help.sap.com/docs/SAP_S4HANA_ON-PREMISE/7340a09096454b7abf4379f926a21567/be98e393c8a94a2aa260ba1db235b5d1.html","engine":"fetch","unsupportedFeatures":{}} 2024-11-18 20:33:15 debug [ScrapeURL:]: Executed transformers.