Best Firecrawl methods for large recurring tasks
I have another question if you @Gaurav Chadha don't mind.
Without addressing the appropriateness of such tasks, I'd like to do the following using Firecrawl -- 1. do a search for all bid publishing sites for one service 2. do a scrape of each site, extracting bids with a "Current" or "Open" criteria (dealing with pagination if present) 3. do a scrape of each bid, extracting a known schema for each bid What is the most cost effective way to do this? Doing step 1 every week, and steps 2 and 3 every day. It seams the
Without addressing the appropriateness of such tasks, I'd like to do the following using Firecrawl -- 1. do a search for all bid publishing sites for one service 2. do a scrape of each site, extracting bids with a "Current" or "Open" criteria (dealing with pagination if present) 3. do a scrape of each bid, extracting a known schema for each bid What is the most cost effective way to do this? Doing step 1 every week, and steps 2 and 3 every day. It seams the
extract method is more expensive, but is it cheaper for large jobs. Steps 2 and 3 are pretty "deterministic" without using an AI model, but when is using AI worth the extra cost?1 Reply
Hi @Robin Mattern you can use cron job instead of doing these tasks each week to be more efficient and avoid any manual work.
refer to the #3 - https://www.firecrawl.dev/blog/n8n-web-scraping-workflow-templates example
for step1 - use
/search query with category
for step 2 and 3 /scrape as discussed here - https://discord.com/channels/1226707384710332458/1418585477342167072
yeah, extract is much better for larger jobs and as it comes with AI LLM it's much more efficient, and it comes with 500k token initially https://docs.firecrawl.dev/features/extractFirecrawl
Web Scraping with n8n: 8 Powerful Workflow Templates
Learn how to build robust web scraping automations with n8n workflow templates that extract data from any website without coding using Firecrawl's AI-powered engine.