F
Firecrawl•5mo ago
fffff

Firecrawl API stopped working when using proxy – anyone else experiencing this?

Hi everyone, I've been using the Firecrawl API for a while now, and up until today, everything was working fine when using a proxy. However, today I started getting SSL-related errors like: >SSLError: HTTPSConnectionPool(host='api.firecrawl.dev', port=443): Max retries exceeded with url: /v1/scrape (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1006)'))) As soon as I disable the proxy , it works perfectly fine again. My questions: Has anyone else experienced similar issues recently with Firecrawl + proxy? Is there any known change on Firecrawl’s side that might affect proxy usage? Are there any workarounds or settings I can use to make it work with a proxy ?
2 Replies
Julian Flieller
Julian Flieller•5mo ago
Hey @fffff are you somehow setting your own proxy or using the stealth / basic option provided by firecrawl? Just tested with the stealth option and it works perfectly fine for me. Also works for me with setting a location. Feel free to DM me so we can sort it out together, happy to help! ---- @Moderator btw the API reference for location.country. link @ https://docs.firecrawl.dev/features/proxies redirects to a localhost:3000 link. and the provided example didnt work me, but the following worked: 
from firecrawl import FirecrawlApp
from pydantic import BaseModel
from typing import List, Optional

class LocationConfig(BaseModel):
    country: str
    languages: Optional[List[str]] = None

app = FirecrawlApp(api_key="YOUR API KEY")

content = app.scrape_url(
    "https://www.firecrawl.dev/", 
    proxy="stealth", 
    formats=['markdown'],
    location=LocationConfig(
        country="US",
        languages=["en-US"]
    )
)

print(content)
from firecrawl import FirecrawlApp
from pydantic import BaseModel
from typing import List, Optional

class LocationConfig(BaseModel):
    country: str
    languages: Optional[List[str]] = None

app = FirecrawlApp(api_key="YOUR API KEY")

content = app.scrape_url(
    "https://www.firecrawl.dev/", 
    proxy="stealth", 
    formats=['markdown'],
    location=LocationConfig(
        country="US",
        languages=["en-US"]
    )
)

print(content)
the provided example in the docs, will produce the following error: 
scrape_params['location'] = location.dict(exclude_none=True)         
AttributeError: 'dict' object has no attribute 'dict'
scrape_params['location'] = location.dict(exclude_none=True)         
AttributeError: 'dict' object has no attribute 'dict'
probably due to .dict() being called on a plain dict in the example.
mogery
mogery•5mo ago
cc @rafaelmiller on the Python SDK end

Did you find this page helpful?