CA
Crawlee & Apify•12mo ago
conscious-sapphire

New Error on Long-Running Actor

Starting this afternoon, the Logs of one of my daily actors (which has run successfully every day for the past 30 days), are being flooded with this message: 
2024-06-13T04:28:54.490Z WARN  ApifyClient: API request failed 4 times. Max attempts: 9.
2024-06-13T04:28:54.492Z Cause:ApifyApiError: You have exceeded the rate limit of 30 requests per second
2024-06-13T04:28:54.495Z   clientMethod: RequestQueueClient.get
2024-06-13T04:28:54.497Z   statusCode: 429
2024-06-13T04:28:54.498Z   type: rate-limit-exceeded
2024-06-13T04:28:54.500Z   attempt: 4
2024-06-13T04:28:54.504Z   httpMethod: get
2024-06-13T04:28:54.505Z   path: /v2/request-queues/tMiFOqCqVhLTL7QSK
2024-06-13T04:28:54.507Z   stack:
2024-06-13T04:28:54.509Z     at makeRequest (/home/myuser/node_modules/apify-client/dist/http_client.js:184:30)
2024-06-13T04:28:54.510Z     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2024-06-13T04:28:54.512Z     at async RequestQueueClient._get (/home/myuser/node_modules/apify-client/dist/base/resource_client.js:25:30)
2024-06-13T04:28:54.514Z     at async RequestQueue.open (/home/myuser/node_modules/@crawlee/core/storages/request_provider.js:614:34)
2024-06-13T04:28:54.516Z     at async PuppeteerCrawler.getRequestQueue (/home/myuser/node_modules/@crawlee/basic/internals/basic-crawler.js:595:51)
2024-06-13T04:28:54.518Z     at async PuppeteerCrawler.addRequests (/home/myuser/node_modules/@crawlee/basic/internals/basic-crawler.js:612:30)
2024-06-13T04:28:54.490Z WARN  ApifyClient: API request failed 4 times. Max attempts: 9.
2024-06-13T04:28:54.492Z Cause:ApifyApiError: You have exceeded the rate limit of 30 requests per second
2024-06-13T04:28:54.495Z   clientMethod: RequestQueueClient.get
2024-06-13T04:28:54.497Z   statusCode: 429
2024-06-13T04:28:54.498Z   type: rate-limit-exceeded
2024-06-13T04:28:54.500Z   attempt: 4
2024-06-13T04:28:54.504Z   httpMethod: get
2024-06-13T04:28:54.505Z   path: /v2/request-queues/tMiFOqCqVhLTL7QSK
2024-06-13T04:28:54.507Z   stack:
2024-06-13T04:28:54.509Z     at makeRequest (/home/myuser/node_modules/apify-client/dist/http_client.js:184:30)
2024-06-13T04:28:54.510Z     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2024-06-13T04:28:54.512Z     at async RequestQueueClient._get (/home/myuser/node_modules/apify-client/dist/base/resource_client.js:25:30)
2024-06-13T04:28:54.514Z     at async RequestQueue.open (/home/myuser/node_modules/@crawlee/core/storages/request_provider.js:614:34)
2024-06-13T04:28:54.516Z     at async PuppeteerCrawler.getRequestQueue (/home/myuser/node_modules/@crawlee/basic/internals/basic-crawler.js:595:51)
2024-06-13T04:28:54.518Z     at async PuppeteerCrawler.addRequests (/home/myuser/node_modules/@crawlee/basic/internals/basic-crawler.js:612:30)
The Actor still seems to be working properly: results are being saved to the dataset, but first I got an email alert that my scheduled task failed to run due to a misconfiguration. I haven't changed the configuration in a while, so I'm not sure how that's possible. Anybody know how I can debug this?
2 Replies
Marco
Marco•12mo ago
Hello! Could you post the IDs of the scheduled task and the run with the warnings, please?
Pepa J
Pepa J•12mo ago
Hi @kennysmithnanic . This is just a warning about you are reaching the limitation of 30requests/sec on the platfrom. It could be a lot of things for example pushing too much data into dataset one-item-by-one - each push is usually one requests to the Platfrom. This limit is across all the Actors, so it also might happened if you ran several actors at once, as it is easirer to reach this limits this way.

Did you find this page helpful?