Broken Links Checker only returns 20 requests
Hi!
When running Broken Links Checker actor by Jan Čurn in Apify, it only returns 20 results instead of all links on the website. Is there possibly a setting that I'm missing? The run ID is WpXfO0LrXg8GestHv.
Thank you in advance for any help you could offer!
Michael
4 Replies
harsh-harlequin•2y ago
Hey! I checked your run and the site and it has over 50 links but only 20 are unique, so the Actor seems to be working correctly. You can choose to crawl the subdomains as well to increase the number of links.
optimistic-goldOP•2y ago
Thank you for your reply @lemurio!
I reran the crawl to include subdomains and it expanded the number of scanned pages. However, it did not crawl every available webpage. Only 10% of pages were crawled in one directory, even though other web pages contain broken links. Pages in other directories containing broken links are not listed in the logs. When I enter a specific URL from one of those pages in the directory in the Website URL textbox (Run ID 5lejFqh56vcW6unny), it finds the broken links on the page. I've been loving Ampify and I'm excited to make it my long-term solution to website crawling, but I want to make sure that this Actor is an accurate, working solution. Thank you again for your time and help! Michael I will make a few adjustments to the website and run another scan. I think I understand why specific directories are being overlooked. Okay! I added a site map, and all pages appear to be crawled! The only hiccup I'm experiencing are false positives on active external links that throw the following warning: WARN Page failed 1 times, giving up. Is there a possible to setting prevent this from giving up after one attempt? Thank you again!
I reran the crawl to include subdomains and it expanded the number of scanned pages. However, it did not crawl every available webpage. Only 10% of pages were crawled in one directory, even though other web pages contain broken links. Pages in other directories containing broken links are not listed in the logs. When I enter a specific URL from one of those pages in the directory in the Website URL textbox (Run ID 5lejFqh56vcW6unny), it finds the broken links on the page. I've been loving Ampify and I'm excited to make it my long-term solution to website crawling, but I want to make sure that this Actor is an accurate, working solution. Thank you again for your time and help! Michael I will make a few adjustments to the website and run another scan. I think I understand why specific directories are being overlooked. Okay! I added a site map, and all pages appear to be crawled! The only hiccup I'm experiencing are false positives on active external links that throw the following warning: WARN Page failed 1 times, giving up. Is there a possible to setting prevent this from giving up after one attempt? Thank you again!
harsh-harlequin•2y ago
Hi! Try to write in the Issues tab of the Actor and request this feature to directly contact the developer!
optimistic-goldOP•2y ago
Awesome! Thanks again for your help!