Crawl gets blocked on sites like MSNBC
Is there a work around to the sites like MSNBC being blocked? I get a 403 error when trying to reach them.
2 Replies
You could try subbing the playwright-service w/ - https://hub.docker.com/r/trieve/puppeteer-service-ts
Uses https://ulixee.org/docs/hero instead of playwright which has a shot of getting around their detector
Hard to know for sure tho