F
Firecrawl•5mo ago
Ramon

Request Timeout in Python SDK uses milliseconds but should be seconds?

The scrape_url method in the python SDK expects timeout to be passed in milliseconds and passes it to the requests library without unit conversion. 
class FirecrawlApp:
    def scrape_url(...):
        ...
        if timeout:
            scrape_params['timeout'] = timeout
        ...
        response = requests.post(
          f'{self.api_url}/v1/scrape',
          headers=headers,
          json=scrape_params,
          timeout=(timeout + 5000 if timeout else None)
        )
class FirecrawlApp:
    def scrape_url(...):
        ...
        if timeout:
            scrape_params['timeout'] = timeout
        ...
        response = requests.post(
          f'{self.api_url}/v1/scrape',
          headers=headers,
          json=scrape_params,
          timeout=(timeout + 5000 if timeout else None)
        )
Other methods delegate to the _post_request method which works similarly: 
    def _post_request(
            self,
            url: str,
            data: Dict[str, Any],
            headers: Dict[str, str],
            retries: int = 3,
            backoff_factor: float = 0.5) -> requests.Response:
        """
        Make a POST request with retries.

        Args:
            url (str): The URL to send the POST request to.
            data (Dict[str, Any]): The JSON data to include in the POST request.
            headers (Dict[str, str]): The headers to include in the POST request.
            retries (int): Number of retries for the request.
            backoff_factor (float): Backoff factor for retries.

        Returns:
            requests.Response: The response from the POST request.

        Raises:
            requests.RequestException: If the request fails after the specified retries.
        """
        for attempt in range(retries):
            response = requests.post(url, headers=headers, json=data, timeout=((data["timeout"] + 5000) if "timeout" in data else None))
            if response.status_code == 502:
                time.sleep(backoff_factor * (2 ** attempt))
            else:
                return response
        return response
    def _post_request(
            self,
            url: str,
            data: Dict[str, Any],
            headers: Dict[str, str],
            retries: int = 3,
            backoff_factor: float = 0.5) -> requests.Response:
        """
        Make a POST request with retries.

        Args:
            url (str): The URL to send the POST request to.
            data (Dict[str, Any]): The JSON data to include in the POST request.
            headers (Dict[str, str]): The headers to include in the POST request.
            retries (int): Number of retries for the request.
            backoff_factor (float): Backoff factor for retries.

        Returns:
            requests.Response: The response from the POST request.

        Raises:
            requests.RequestException: If the request fails after the specified retries.
        """
        for attempt in range(retries):
            response = requests.post(url, headers=headers, json=data, timeout=((data["timeout"] + 5000) if "timeout" in data else None))
            if response.status_code == 502:
                time.sleep(backoff_factor * (2 ** attempt))
            else:
                return response
        return response
The requests library is expecting the timeout parameter to be in secnods rather than milliseconds. Is this a bug in the current SDK? My code stalls from time to time, and I think this could be the reason.
1 Reply
mogery
mogery•5mo ago
@rafaelmiller

Did you find this page helpful?