Topics

Firecrawl•3w ago

/crawl robots.txt

Does /crawl respect robots.tx Crawl-delay?

2 Replies

micah.stairs•3w ago

Yep!

rno321OP•3w ago

Someone else told me it only respects allow/disallow params, not crawl-delay.

8.8KMembers

View on Discord

Did you find this page helpful?