enqueue urls / request queue not being unique
I'm seeing a lot of the same exact URL's being ran twice? Any ideas?
2 Replies
View post on community site
This post has been pushed to the community knowledgebase. Any replies in this thread will be synced to the community site.
Apify Community
exotic-emerald•7mo ago
Hi. Hm...that is not the default behavior 🤔.
The equality of two requests is determined via comparison of their
uniqueKey properties. By changing this property value, you could be able to process the same URLs multiple times (if their uniqueKey values were unique ofc). Do you explicitly set this property?
See this.Request | API | Crawlee · Build reliable crawlers. Fast.
Crawlee helps you build and maintain your crawlers. It's open source, but built by developers who scrape millions of pages every day for a living.