Issue while scraping a website: code='CRAWL_DENIAL'
- I created a crawl job with the following params:
crawl_job = client.start_crawl(url='https://jenis.com/blogs/scoop-shops/alexandria-va', limit=1, scrape_options={'format': 'markdown', 'timeout': 120000, 'exclude_tags': excluded_tags_from_html(), 'proxy': 'auto', 'only_main_content': False}, max_discovery_depth=20)
- Then after that I ran the following snippet
job_id = crawl_job.id
print(f"Started Firecrawl job with ID: {job_id}")
status = client.get_crawl_status(job_id)
print(f"Crawl Status: {status}")
errors = client.get_crawl_errors(job_id)
print(f"Crawl Errors: {errors}")
And the output was something like this:
Started Firecrawl job with ID:
019b8ee2-611b-7612-afea-bc1e1fd8a2a4
Crawl Status: status='completed' total=0 completed=0 credits_used=0 expires_at=datetime.datetime(2026, 1, 6, 15, 59, 25, tzinfo=TzInfo(0)) next=None data=[]
Crawl Errors: errors=[CrawlError(id='019b8ee2-62f9-71f1-96e0-0b87aaa574b4', timestamp=datetime.datetime(2026, 1, 5, 15, 59, 20, 189000, tzinfo=TzInfo(0)), url='https://jenis.com/blogs/scoop-shops/alexandria-va', code='CRAWL_DENIAL', error='URL contains section anchor (#)')] robots_blocked=[]
In the output you can find the job id of the crawl, that may help to track the issue. Also the data associated with status is empty.
crawl_job = client.start_crawl(url='https://jenis.com/blogs/scoop-shops/alexandria-va', limit=1, scrape_options={'format': 'markdown', 'timeout': 120000, 'exclude_tags': excluded_tags_from_html(), 'proxy': 'auto', 'only_main_content': False}, max_discovery_depth=20)
- Then after that I ran the following snippet
job_id = crawl_job.id
print(f"Started Firecrawl job with ID: {job_id}")
status = client.get_crawl_status(job_id)
print(f"Crawl Status: {status}")
errors = client.get_crawl_errors(job_id)
print(f"Crawl Errors: {errors}")
And the output was something like this:
Started Firecrawl job with ID:
019b8ee2-611b-7612-afea-bc1e1fd8a2a4
Crawl Status: status='completed' total=0 completed=0 credits_used=0 expires_at=datetime.datetime(2026, 1, 6, 15, 59, 25, tzinfo=TzInfo(0)) next=None data=[]
Crawl Errors: errors=[CrawlError(id='019b8ee2-62f9-71f1-96e0-0b87aaa574b4', timestamp=datetime.datetime(2026, 1, 5, 15, 59, 20, 189000, tzinfo=TzInfo(0)), url='https://jenis.com/blogs/scoop-shops/alexandria-va', code='CRAWL_DENIAL', error='URL contains section anchor (#)')] robots_blocked=[]
In the output you can find the job id of the crawl, that may help to track the issue. Also the data associated with status is empty.
