Exclude subdomain from being scraped
Is it possible to exclude a subdomain when scraping a website?
I have a client site I would like to scrape, but they recently added a subdomain which list specific products/services not directly related to their main service.
e.g
www.site-name.com (main)
www.subdomain.site-name.com (subdomain - I want to exclude)
I have a client site I would like to scrape, but they recently added a subdomain which list specific products/services not directly related to their main service.
e.g
www.site-name.com (main)
www.subdomain.site-name.com (subdomain - I want to exclude)
