Split crawler and scraper into 2 container
I want to split crawler & scraper container, scraper container can call crawler container to get html. 2 container call by gRPC or JSON RPC. Purpose for this, I want 2 process is independent (crawler container can cache or store own data)
Thank you!
4 Replies
judicial-coral•3y ago
This is very possible, but what have you already tried? We can help you with a problem you're facing, but not create an entire solution for you.
absent-sapphireOP•3y ago
I need to run PupperteerCrawler in the gRPC server and scraper container to get data from Crawler vi gRPC protocal
conscious-sapphire•3y ago
This is not specific to Crawlee in any way. Just send the URLs to the seconds container
absent-sapphireOP•3y ago
thank you