Minh Thông - Hi guys, could you give me Firecra...

Hi guys, could you give me Firecrawl User-agent name ? So I can allow it in robots.txt. Thanks!

2 Replies

amplitudes•4w ago

Hey @Minh Thông, our crawler checks for FireCrawlAgent in robots.txt

Minh ThôngOP•4w ago

Thanks you!

Firecrawl Join

Join builders, developers, users and turn any website into LLM-ready data, enabling developers to power their AI applications with clean, structured information crawled from the web.

12KMembers

View on Discord

Gaming

Programming

Minh Thông - Hi guys, could you give me Firecra...

Did you find this page helpful?