@Conner If there are components, then this could be a race condition. try adding a wait to your pup

IIan @Conner If there are components, then this could be a race condition. try addin...

C

Conner•5/22/24, 9:49 PM

Awesome thanks... what should I wait for?
Because I screenshot the page before doing __doPostBack and everything looks loaded in

I

IanOP•5/22/24, 9:50 PM

there might be a "scripts loaded" event in puppeteer, or maybe you need to turn on JS support. not 100% sure.

I

IanOP•5/22/24, 10:13 PM

@Conner You could use

Page.waitForFunction()

Page.waitForFunction()

Page.waitForFunction()

Page.waitForFunction() to return a truthy value https://pptr.dev/api/puppeteer.page.waitforfunction

Page.waitForFunction() method | Puppeteer

Waits for the provided function, pageFunction, to return a truthy value when evaluated in the page's context.

IIan @Conner You could use `Page.waitForFunction()` to return a truthy value https://...

C

Conner•5/22/24, 10:13 PM

I'm evaling other code before that to be clear
Basic querySelector for elements on the page

C

Conner•5/22/24, 10:13 PM

Just seems to be that function that's not available

I

IanOP•5/22/24, 10:15 PM

the elements will exist on thet page from the start since they're rendered on the server, however, the JS that adds

__doPostBack

__doPostBack

__doPostBack

__doPostBack will not be instantaneous.

C

Conner•5/22/24, 10:15 PM

Ah that makes sense

C

Conner•5/22/24, 10:15 PM

I'll try a waitForFunction thanks

I

IanOP•5/22/24, 10:15 PM

you could also just try a 5 second timeout to see if that works since it will confirm it's a race condition.

I

IanOP•5/22/24, 10:16 PM

it most likely works locally because the page network response times are faster locally.

I

IanOP•5/22/24, 10:16 PM

wait, did you try locally but pointing at the remote page?

C

Conner•5/22/24, 10:16 PM

Yes locally pointing at the remote page

C

Conner•5/22/24, 10:16 PM

Same code locally and in cloudflare, a remote site I'm scraping

I

IanOP•5/22/24, 10:17 PM

okay.

I

IanOP•5/22/24, 10:17 PM

Ensure you have

waitUntil

waitUntil

set to

load

load

load

load

await page.goto('https://google.com', { waitUntil: 'load' });

await page.goto('https://google.com', { waitUntil: 'load' });

await page.goto('https://google.com', { waitUntil: 'load' });

await page.goto('https://google.com', { waitUntil: 'load' });

C

Conner•5/22/24, 10:18 PM

I do yeah
I'll try the waitForFunction gimme a couple minutes and I'll lyk

C

Conner•5/22/24, 10:18 PM

await page.waitForFunction(() => typeof window.__doPostBack === 'function');

await page.waitForFunction(() => typeof window.__doPostBack === 'function');

await page.waitForFunction(() => typeof window.__doPostBack === 'function');

await page.waitForFunction(() => typeof window.__doPostBack === 'function');

I

IanOP•5/22/24, 10:18 PM

https://www.webshare.io/academy-article/puppeteer-wait-for-page-to-load this article explains the various waitUntil options. You can wait for network idle, for example.

C

Conner•5/22/24, 10:20 PM

I do have

networkidle0

networkidle0

networkidle0

networkidle0 actually yeah

C

Conner•5/22/24, 10:23 PM

@Ian The waitForFunction didn't work,

TimeoutError: waiting for function failed: timeout 30000ms exceeded

TimeoutError: waiting for function failed: timeout 30000ms exceeded

TimeoutError: waiting for function failed: timeout 30000ms exceeded

TimeoutError: waiting for function failed: timeout 30000ms exceeded

C

Conner•5/22/24, 10:23 PM

Any chance cloudflare's browser might not load the aspx?

I

IanOP•5/22/24, 10:23 PM

okay, so the JS is never running.

CConner Any chance cloudflare's browser might not load the aspx?

I

IanOP•5/22/24, 10:24 PM

nothing about ASPX pages that make them special.

I

IanOP•5/22/24, 10:24 PM

It could be a config difference.

C

Conner•5/22/24, 10:24 PM

Could it be an https thing?

C

Conner•5/22/24, 10:25 PM

Does the ASPX page load the __doPostBack over https or http

C

Conner•5/22/24, 10:25 PM

async custom_launch(endpoint: BrowserWorker): Promise<Browser> {
        const res = await endpoint.fetch('/v1/acquire');
        const status = res.status;
        const text = await res.text();
        if (status !== 200) {
            throw new Error(
                `Unabled to create new browser: code: ${status}: message: ${text}`
            );
        }
        // Got a 200, so response text is actually an AcquireResponse
        const response: { sessionId: string } = JSON.parse(text);
        const transport = await WorkersWebSocketTransport.create(
            endpoint,
            response.sessionId
        );
        return puppeteer.connect({ transport, sessionId: response.sessionId, ignoreHTTPSErrors: true, defaultViewport: { width: 1920, height: 1080 } });
    }

async custom_launch(endpoint: BrowserWorker): Promise<Browser> {
        const res = await endpoint.fetch('/v1/acquire');
        const status = res.status;
        const text = await res.text();
        if (status !== 200) {
            throw new Error(
                `Unabled to create new browser: code: ${status}: message: ${text}`
            );
        }
        // Got a 200, so response text is actually an AcquireResponse
        const response: { sessionId: string } = JSON.parse(text);
        const transport = await WorkersWebSocketTransport.create(
            endpoint,
            response.sessionId
        );
        return puppeteer.connect({ transport, sessionId: response.sessionId, ignoreHTTPSErrors: true, defaultViewport: { width: 1920, height: 1080 } });
    }

async custom_launch(endpoint: BrowserWorker): Promise<Browser> {
        const res = await endpoint.fetch('/v1/acquire');
        const status = res.status;
        const text = await res.text();
        if (status !== 200) {
            throw new Error(
                `Unabled to create new browser: code: ${status}: message: ${text}`
            );
        }
        // Got a 200, so response text is actually an AcquireResponse
        const response: { sessionId: string } = JSON.parse(text);
        const transport = await WorkersWebSocketTransport.create(
            endpoint,
            response.sessionId
        );
        return puppeteer.connect({ transport, sessionId: response.sessionId, ignoreHTTPSErrors: true, defaultViewport: { width: 1920, height: 1080 } });
    }

async custom_launch(endpoint: BrowserWorker): Promise<Browser> {
        const res = await endpoint.fetch('/v1/acquire');
        const status = res.status;
        const text = await res.text();
        if (status !== 200) {
            throw new Error(
                `Unabled to create new browser: code: ${status}: message: ${text}`
            );
        }
        // Got a 200, so response text is actually an AcquireResponse
        const response: { sessionId: string } = JSON.parse(text);
        const transport = await WorkersWebSocketTransport.create(
            endpoint,
            response.sessionId
        );
        return puppeteer.connect({ transport, sessionId: response.sessionId, ignoreHTTPSErrors: true, defaultViewport: { width: 1920, height: 1080 } });
    }

I had to do this a while back for an unrelated page. Added the

ignoreHTTPSErrors: true

ignoreHTTPSErrors: true

ignoreHTTPSErrors: true

ignoreHTTPSErrors: true. Instead of default

puppeteer.launch

puppeteer.launch

puppeteer.launch

puppeteer.launch

I

IanOP•5/22/24, 10:25 PM

if you've got mixed https and http resources, it might be refusing to load cross origin.

I

IanOP•5/22/24, 10:26 PM

yeah, you might need

ignoreHTTPSErrors

ignoreHTTPSErrors

ignoreHTTPSErrors

ignoreHTTPSErrors enabled.

C

Conner•5/22/24, 10:26 PM

Makes sense alright I'll give the

custom_launch

custom_launch

custom_launch

custom_launch a try

I

IanOP•5/22/24, 10:34 PM

You don't need that whole function, just when you call

puppeteer.connect

puppeteer.connect

puppeteer.connect

puppeteer.connect pass it the

ignoreHTTPSErrors: true

ignoreHTTPSErrors: true

ignoreHTTPSErrors: true

ignoreHTTPSErrors: true

IIan You don't need that whole function, just when you call `puppeteer.connect` pass ...

C

Conner•5/22/24, 10:51 PM

Still fails with

TimeoutError: waiting for function failed: timeout 30000ms exceeded

TimeoutError: waiting for function failed: timeout 30000ms exceeded

TimeoutError: waiting for function failed: timeout 30000ms exceeded

TimeoutError: waiting for function failed: timeout 30000ms exceeded

C

Conner•5/22/24, 10:51 PM

Any other ideas
Odd it works from a local puppeteer but not in cloudflare

C

Conner•5/22/24, 10:58 PM

Also @Ian it looks like

__doPostBack

__doPostBack

__doPostBack

__doPostBack is defined inline in the javascript of the page, not loaded externally

<script type="text/javascript">
    //<![CDATA[
    var theForm = document.forms['form1'];
    if (!theForm) {
        theForm = document.form1;
    }
    function __doPostBack(eventTarget, eventArgument) {
        if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
            theForm.__EVENTTARGET.value = eventTarget;
            theForm.__EVENTARGUMENT.value = eventArgument;
            theForm.submit();
        }
    }
    //]]>
</script>

<script type="text/javascript">
    //<![CDATA[
    var theForm = document.forms['form1'];
    if (!theForm) {
        theForm = document.form1;
    }
    function __doPostBack(eventTarget, eventArgument) {
        if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
            theForm.__EVENTTARGET.value = eventTarget;
            theForm.__EVENTARGUMENT.value = eventArgument;
            theForm.submit();
        }
    }
    //]]>
</script>

<script type="text/javascript">
    //<![CDATA[
    var theForm = document.forms['form1'];
    if (!theForm) {
        theForm = document.form1;
    }
    function __doPostBack(eventTarget, eventArgument) {
        if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
            theForm.__EVENTTARGET.value = eventTarget;
            theForm.__EVENTARGUMENT.value = eventArgument;
            theForm.submit();
        }
    }
    //]]>
</script>

<script type="text/javascript">
    //<![CDATA[
    var theForm = document.forms['form1'];
    if (!theForm) {
        theForm = document.form1;
    }
    function __doPostBack(eventTarget, eventArgument) {
        if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
            theForm.__EVENTTARGET.value = eventTarget;
            theForm.__EVENTARGUMENT.value = eventArgument;
            theForm.submit();
        }
    }
    //]]>
</script>

C

Conner•5/22/24, 11:00 PM

Could it be some anti scraping thing? It's an old website so I wouldn't imagine so

I

IanOP•5/22/24, 11:17 PM

okay then that script is there right from the start; no load time.

I

IanOP•5/22/24, 11:18 PM

This makes me wonder if scripts are disabled, so that JS never gets evaluated.

I

IanOP•5/22/24, 11:23 PM

@Conner could it be this https://stackoverflow.com/a/55766824/46914 ?

Stack Overflow

How can I execute a __doPostBack function in puppeteer?

I am trying to scrape a web page that requires clicking on a page button through a doPostBack function. I have tried the following code in the chrome console.

javascript:doPostBack('ctl00$

IIan @Conner could it be this https://stackoverflow.com/a/55766824/46914 ?

C

Conner•5/23/24, 5:05 AM

Yeah I tried calling .click on the a tag, but it didn't do anything - I'm assuming because it's href is

javascript:__doPostBack(...)

javascript:__doPostBack(...)

javascript:__doPostBack(...)

javascript:__doPostBack(...), which is still undefined

IIan This makes me wonder if scripts are disabled, so that JS never gets evaluated.

C

Conner•5/23/24, 5:06 AM

If scripts were disabled, wouldn't my other .evaluate calls fail?

C

Conner•5/23/24, 5:06 AM

It has to be some difference between cloudflare puppeteer/chromium and normal puppeteer/chromium right? Because that's the only difference when I run it locally

CConner If scripts were disabled, wouldn't my other .evaluate calls fail?

I

IanOP•5/23/24, 5:06 AM

Not necessarily. it literally might be ignoring

<script>

<script>

tag contents, but the javascript interpreter is still running, hence why

.evaluate()

.evaluate()

.evaluate()

.evaluate() would work.

CConner It has to be some difference between cloudflare puppeteer/chromium and normal pu...

I

IanOP•5/23/24, 5:07 AM

Yes, that definitely could be it. are you sure you're running the same version?

I

IanOP•5/23/24, 5:08 AM

also, are there any flags that cloudflare might enable/disable?

C

Conner•5/23/24, 5:09 AM

The script for this is in the body if that makes any difference

<body>
    <form method="post" action="./action.aspx" onkeypress="javascript:return WebForm_FireDefaultButton(event, 'criteria_basic_btn_search')" id="form1">
        <div class="aspNetHidden">
            <input type="hidden" name="__EVENTTARGET" id="__EVENTTARGET" value=""/>
            <input type="hidden" name="__EVENTARGUMENT" id="__EVENTARGUMENT" value=""/>
            <input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="...e"/>
        </div>
        <script type="text/javascript">
            //<![CDATA[
            var theForm = document.forms['form1'];
            if (!theForm) {
                theForm = document.form1;
            }
            function __doPostBack(eventTarget, eventArgument) {
                if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
                    theForm.__EVENTTARGET.value = eventTarget;
                    theForm.__EVENTARGUMENT.value = eventArgument;
                    theForm.submit();
                }
            }
            //]]>
        </script>
        ...

<body>
    <form method="post" action="./action.aspx" onkeypress="javascript:return WebForm_FireDefaultButton(event, 'criteria_basic_btn_search')" id="form1">
        <div class="aspNetHidden">
            <input type="hidden" name="__EVENTTARGET" id="__EVENTTARGET" value=""/>
            <input type="hidden" name="__EVENTARGUMENT" id="__EVENTARGUMENT" value=""/>
            <input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="...e"/>
        </div>
        <script type="text/javascript">
            //<![CDATA[
            var theForm = document.forms['form1'];
            if (!theForm) {
                theForm = document.form1;
            }
            function __doPostBack(eventTarget, eventArgument) {
                if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
                    theForm.__EVENTTARGET.value = eventTarget;
                    theForm.__EVENTARGUMENT.value = eventArgument;
                    theForm.submit();
                }
            }
            //]]>
        </script>
        ...

<body>
    <form method="post" action="./action.aspx" onkeypress="javascript:return WebForm_FireDefaultButton(event, 'criteria_basic_btn_search')" id="form1">
        <div class="aspNetHidden">
            <input type="hidden" name="__EVENTTARGET" id="__EVENTTARGET" value=""/>
            <input type="hidden" name="__EVENTARGUMENT" id="__EVENTARGUMENT" value=""/>
            <input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="...e"/>
        </div>
        <script type="text/javascript">
            //<![CDATA[
            var theForm = document.forms['form1'];
            if (!theForm) {
                theForm = document.form1;
            }
            function __doPostBack(eventTarget, eventArgument) {
                if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
                    theForm.__EVENTTARGET.value = eventTarget;
                    theForm.__EVENTARGUMENT.value = eventArgument;
                    theForm.submit();
                }
            }
            //]]>
        </script>
        ...

<body>
    <form method="post" action="./action.aspx" onkeypress="javascript:return WebForm_FireDefaultButton(event, 'criteria_basic_btn_search')" id="form1">
        <div class="aspNetHidden">
            <input type="hidden" name="__EVENTTARGET" id="__EVENTTARGET" value=""/>
            <input type="hidden" name="__EVENTARGUMENT" id="__EVENTARGUMENT" value=""/>
            <input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="...e"/>
        </div>
        <script type="text/javascript">
            //<![CDATA[
            var theForm = document.forms['form1'];
            if (!theForm) {
                theForm = document.form1;
            }
            function __doPostBack(eventTarget, eventArgument) {
                if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
                    theForm.__EVENTTARGET.value = eventTarget;
                    theForm.__EVENTARGUMENT.value = eventArgument;
                    theForm.submit();
                }
            }
            //]]>
        </script>
        ...