CA
Crawlee & Apify•3y ago
sensitive-blue

Firefox, PlaywrightCrawler, SSL_ERROR_BAD_CERT_DOMAIN error

One of the pages I want to scrape with PlaywrightCrawler returns the SSL_ERROR_BAD_CERT_DOMAIN error. I can reproduce this error when I open this URL in Firefox/Chrome - I see the browser shows the prompt with the warning and asks "...do you want to proceed?" So the error is from the browser, not from Crawlee/Playwright... But... Firefox has so many flags/settings... may be I can somehow set the flag "accept all certificates"?
2 Replies
sensitive-blue
sensitive-blueOP•3y ago
UPDATE: I see ignoreHTTPSErrors in Playwright https://playwright.dev/docs/api/class-testoptions#test-options-ignore-https-errors In the code like like this: 
const crawler = new PlaywrightCrawler({

    autoscaledPoolOptions: {
    ...
    },


    headless: true,
    browserPoolOptions: {
        ...

        prePageCreateHooks: [
            (pageId, browserController, pageOptions) => {
               ...
            },
        ],
    },
    ...

    launchContext: {
            ...
        useIncognitoPages: true,
        launcher: myLauncher
    },
const crawler = new PlaywrightCrawler({

    autoscaledPoolOptions: {
    ...
    },


    headless: true,
    browserPoolOptions: {
        ...

        prePageCreateHooks: [
            (pageId, browserController, pageOptions) => {
               ...
            },
        ],
    },
    ...

    launchContext: {
            ...
        useIncognitoPages: true,
        launcher: myLauncher
    },
How to add the ignoreHTTPSErrors flag?
TestOptions | Playwright
Playwright Test provides many options to configure test environment, [Browser], [BrowserContext] and more.
sensitive-blue
sensitive-blueOP•3y ago

Did you find this page helpful?