Crawlee & Apify•3y ago

Sending requests to API causes timeout for 60 seconds

While using sendRequest causes Reclaiming failed request back to the list or queue. requestHandler timed out after 60 seconds. Also, using PlaywrightCrawler with Proxy. I am using following code (not same but similar):

{
nagivationTimeoutSecs: 300,
async requestHandler({ sendRequest }) {
  async function getData(page: number =   1) {
    const res = await sendRequest({
      url: "https://example.com/api/foo/bar",
      method: "GET",
    })

    if (res.next) await getData(page + 1)
  }

  await getData()
}
}

{
nagivationTimeoutSecs: 300,
async requestHandler({ sendRequest }) {
  async function getData(page: number =   1) {
    const res = await sendRequest({
      url: "https://example.com/api/foo/bar",
      method: "GET",
    })

    if (res.next) await getData(page + 1)
  }

  await getData()
}
}

3 Replies

flat-fuchsiaOP•3y ago

Using got itself instead of using sendRequest will also causes this. Finally I got requestHandlerTimeoutSecs to pass to PlaywrightCrawler's option as workaround.

MEE6•3y ago

@rikusen0335 just advanced to level 1! Thanks for your contributions! 🎉

flat-fuchsiaOP•3y ago

~~But do we not have any way to reset timeout count when any request/crawl happens?~~ this should be another question

Gaming

Programming

Sending requests to API causes timeout for 60 seconds

Did you find this page helpful?