Do you know a dashboard for Crawleee?
I want to monitor all Crawlee crawlers. To do this I looked for a dashboard to control the crawlers.
I only found this distributed web crawler management platform: Crawlab (https://github.com/crawlab-team/crawlab).
Do you know of any others?
8 Replies
eastern-cyan•3y ago
Have you tried the Apify platform? If you deploy your crawlers to the platform, you can run them and monitor them right from the dashboard there!
https://apify.com
The great thing about this solution is that Crawlee was built with the Apify platform in mind, so there is seamless support
Apify
Web Scraping, Data Extraction and Automation · Apify
Apify is your one-stop shop for web scraping, data extraction, and RPA. Automate anything you can do manually in a browser.
exotic-emeraldOP•3y ago
Thanks for your answer, but using sensitive data, I cannot use a Cloud solution. That's why I'm looking for an open source solution and I use Crawlee.
eastern-cyan•3y ago
So, your crawlers will only be run locally?
exotic-emeraldOP•3y ago
Yes, I have a cluster of server where crawlee crawler run.
eastern-cyan•3y ago
How are these servers being hosted?
exotic-emeraldOP•3y ago
These are Raspberry pi's at my house
@LeMoussel just advanced to level 1! Thanks for your contributions! 🎉
eastern-cyan•3y ago
In order to make a dashboard, you'd need a centralized location where you could store data about each crawler. A SQL or Mongo database would be great for this.
You could have a server hosted on one of your rasp pis that serves as an API for storing data about crawlers. When a new crawler is started, a new row could be added to the "crawlers" table by sending a request from the crawler to your API.
Then, you could create some sort of UI service with Electron or React Native that uses your API.
With this custom solution, you could make the dashboard as minimalistic or as robust as you like. You'd have full control. And everything would be hosted locally.