Cloudflare Developers•2y ago

@Conner could it be this https://stackoverflow.com/a/55766824/46914 ?

@Conner could it be this https://stackoverflow.com/a/55766824/46914 ?

Stack Overflow

How can I execute a __doPostBack function in puppeteer?

I am trying to scrape a web page that requires clicking on a page button through a doPostBack function. I have tried the following code in the chrome console.

javascript:doPostBack('ctl00$

© 2026 Hedgehog Software, LLC

More

Communities Docs About Terms Privacy

Ian

IIan @Conner could it be this https://stackoverflow.com/a/55766824/46914 ?

Conner

C

Conner•5/23/24, 5:05 AM

Yeah I tried calling .click on the a tag, but it didn't do anything - I'm assuming because it's href is javascript:__doPostBack(...)javascript:__doPostBack(...), which is still undefined

Ian

IIan This makes me wonder if scripts are disabled, so that JS never gets evaluated.

Conner

C

Conner•5/23/24, 5:06 AM

If scripts were disabled, wouldn't my other .evaluate calls fail?

Conner

C

Conner•5/23/24, 5:06 AM

It has to be some difference between cloudflare puppeteer/chromium and normal puppeteer/chromium right? Because that's the only difference when I run it locally

Conner

CConner If scripts were disabled, wouldn't my other .evaluate calls fail?

Ian

I

IanOP•5/23/24, 5:06 AM

Not necessarily. it literally might be ignoring <script><script> tag contents, but the javascript interpreter is still running, hence why .evaluate().evaluate() would work.

Conner

CConner It has to be some difference between cloudflare puppeteer/chromium and normal pu...

Ian

I

IanOP•5/23/24, 5:07 AM

Yes, that definitely could be it. are you sure you're running the same version?

Ian

I

IanOP•5/23/24, 5:08 AM

also, are there any flags that cloudflare might enable/disable?

Conner

C

Conner•5/23/24, 5:09 AM

The script for this is in the body if that makes any difference

<body>
    <form method="post" action="./action.aspx" onkeypress="javascript:return WebForm_FireDefaultButton(event, 'criteria_basic_btn_search')" id="form1">
        <div class="aspNetHidden">
            <input type="hidden" name="__EVENTTARGET" id="__EVENTTARGET" value=""/>
            <input type="hidden" name="__EVENTARGUMENT" id="__EVENTARGUMENT" value=""/>
            <input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="...e"/>
        </div>
        <script type="text/javascript">
            //<![CDATA[
            var theForm = document.forms['form1'];
            if (!theForm) {
                theForm = document.form1;
            }
            function __doPostBack(eventTarget, eventArgument) {
                if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
                    theForm.__EVENTTARGET.value = eventTarget;
                    theForm.__EVENTARGUMENT.value = eventArgument;
                    theForm.submit();
                }
            }
            //]]>
        </script>
        ...

<body>
    <form method="post" action="./action.aspx" onkeypress="javascript:return WebForm_FireDefaultButton(event, 'criteria_basic_btn_search')" id="form1">
        <div class="aspNetHidden">
            <input type="hidden" name="__EVENTTARGET" id="__EVENTTARGET" value=""/>
            <input type="hidden" name="__EVENTARGUMENT" id="__EVENTARGUMENT" value=""/>
            <input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="...e"/>
        </div>
        <script type="text/javascript">
            //<![CDATA[
            var theForm = document.forms['form1'];
            if (!theForm) {
                theForm = document.form1;
            }
            function __doPostBack(eventTarget, eventArgument) {
                if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
                    theForm.__EVENTTARGET.value = eventTarget;
                    theForm.__EVENTARGUMENT.value = eventArgument;
                    theForm.submit();
                }
            }
            //]]>
        </script>
        ...

Ian

IIan Yes, that definitely could be it. are you sure you're running the same version?

Conner

C

Conner•5/23/24, 5:10 AM

Yep, 17, just like @cloudflare/puppeteer github pages says they use

Ian

IIan also, are there any flags that cloudflare might enable/disable?

Conner

C

Conner•5/23/24, 5:10 AM

And not sure, I couldn't find it online anywhere... any team members have any idea in here?

Ian

I

IanOP•5/23/24, 5:12 AM

Oh I wonder if the //<![CDATA[//<![CDATA[ makes some kind of difference here? I don't think it does.

Ian

I

IanOP•5/23/24, 5:16 AM

@Conner If you visit the site directly, are you able to access __doPostBack__doPostBack ?

Ian

I

IanOP•5/23/24, 5:17 AM

Also, what is the actual puppeteer code you're using for this?

Conner

C

Conner•5/23/24, 5:23 AM

This times out await page.waitForFunction(() => typeof window.__doPostBack === 'function', { timeout: 30000, polling: 100 });await page.waitForFunction(() => typeof window.__doPostBack === 'function', { timeout: 30000, polling: 100 });

In chrome devtools if I do __doPostBack.toString()__doPostBack.toString() I get

function __doPostBack(eventTarget, eventArgument) {
    if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
        theForm.__EVENTTARGET.value = eventTarget;
        theForm.__EVENTARGUMENT.value = eventArgument;
        theForm.submit();
    }
}

function __doPostBack(eventTarget, eventArgument) {
    if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
        theForm.__EVENTTARGET.value = eventTarget;
        theForm.__EVENTARGUMENT.value = eventArgument;
        theForm.submit();
    }
}

As expected

Ian

IIan Oh I wonder if the `//<![CDATA[` makes some kind of difference here? I don't thi...

Conner

C

Conner•5/23/24, 5:24 AM

I don't believe so, I'm scraping the same page on local puppeteer as cloudflare puppeteer haha

Ian

I

IanOP•5/23/24, 5:25 AM

Yeah, I'm just trying to figure out what's different; obviously it's something to do with the config cloudflare is using, or the network itself.

Ian

I

IanOP•5/23/24, 5:26 AM

You might want to try pointing the cloudflare puppeteer at one of those user agent inspectors to see what it says https://www.whatsmyua.info/

What's my user agent?

Detect user-agent, operating system, browser, and device using several libraries, including ua-parser, ua-parser-js, and platform.

Ian

I

IanOP•5/23/24, 5:27 AM

take screenshots of both and compare them to see what the differences are.

Conner

C

Conner•5/23/24, 5:29 AM

I'm already setting user agent

await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36');

await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36');

and I've screenshotted both and they're the same

Ian

IIan Yeah, I'm just trying to figure out what's different; obviously it's something t...

Conner

C

Conner•5/23/24, 5:30 AM

I couldn't find anything on any special config cloudflare has but if you know of anything that'd be awesome

Ian

I

IanOP•5/23/24, 5:30 AM

not that I can see. what about console logs or errors, are you gathering those?

Conner

C

Conner•5/23/24, 5:31 AM

Not gathering any console logs, that's a good idea

Conner

C

Conner•5/23/24, 5:31 AM

My other next thought is maybe inspect the page network requests? Making sure they line up with what I see in devtools in a normal browser?

Ian

I

IanOP•5/23/24, 5:31 AM

https://blog.adamcameron.me/2021/01/listening-to-console-log-of-page-loaded.html

Listening to the console log of a page loaded with Puppeteer

Adam Cameron works out how to listen for a Puppeteer page's log activities.

Conner

CConner My other next thought is maybe inspect the page network requests? Making sure th...

Ian

I

IanOP•5/23/24, 5:32 AM

yeah, though, you already have the JS you need right there baked into the page.

Ian

I

IanOP•5/23/24, 5:32 AM

My current guess is something is crashing and blocking that function from being created.

Ian

I

IanOP•5/23/24, 5:33 AM

@Conner quick question, are you literally calling __doCallBack__doCallBack ? Or clicking the submit button?

Ian

IIan @Conner quick question, are you literally calling `__doCallBack` ? Or clicking t...

Conner

C

Conner•5/23/24, 5:34 AM

I've tried both, neither worked

Conner

C

Conner•5/23/24, 5:34 AM

I also tried running the code inside __doCallBack

Conner

C

Conner•5/23/24, 5:35 AM

function __doPostBack(eventTarget, eventArgument) {
    if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
        theForm.__EVENTTARGET.value = eventTarget;
        theForm.__EVENTARGUMENT.value = eventArgument;
        theForm.submit();
    }
}

function __doPostBack(eventTarget, eventArgument) {
    if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
        theForm.__EVENTTARGET.value = eventTarget;
        theForm.__EVENTARGUMENT.value = eventArgument;
        theForm.submit();
    }
}

When I tried running that code inside, I got something like cannot set property value of undefined

Conner

C

Conner•5/23/24, 5:35 AM

So theForm.__EVENTTARGET is undefined? Which again is weird because the whole page visually loads in fine

Ian

I

IanOP•5/23/24, 5:36 AM

Right because theFormtheForm isn't defined in there, it's defined just outside by

var theForm = document.forms['form1'];

var theForm = document.forms['form1'];

Conner

CConner The script for this is in the body if that makes any difference ``` <body> ...

Conner

C

Conner•5/23/24, 5:36 AM

It's a global variable but yeah I also tried querySelector #form1

Ian

I

IanOP•5/23/24, 5:39 AM

so what's supposed to happen is when a button on the aspx page is clicked it will call __doPostBack__doPostBack and fill in the eventTarget and any eventArgs. These then get filled in as those hidden __EVENTTARGET__EVENTTARGET and __EVENTARGUMENT__EVENTARGUMENT input fields defined above. Those get submitted to the server and that's how it knows what button you pressed so it can run the appropriate event handler code on the server, then send you back a new page render.

Ian

I

IanOP•5/23/24, 5:40 AM

so you should definitely be clicking directly on a button if you can help it to make sure that gets filled out correctly.

Conner

C

Conner•5/23/24, 5:43 AM

In some places 100%

But some I know what I need to fill in so I'm just straight up calling evaluate, for example

pageNumber += 1;

await page.waitForFunction(() => typeof window.__doPostBack === 'function', { timeout: 30000, polling: 100 });

await Promise.all([
    page.waitForNavigation({ waitUntil: 'networkidle0' }),
    page.evaluate(`__doPostBack('gridItems','Page$${pageNumber}')`),
]);

pageNumber += 1;

await page.waitForFunction(() => typeof window.__doPostBack === 'function', { timeout: 30000, polling: 100 });

await Promise.all([
    page.waitForNavigation({ waitUntil: 'networkidle0' }),
    page.evaluate(`__doPostBack('gridItems','Page$${pageNumber}')`),
]);

Conner

C

Conner•5/23/24, 5:44 AM

I'll check console and lyk, I think you're write it might be failing somehow and logging to console

Ian

I

IanOP•5/23/24, 5:46 AM

double check you've got

await

await

everywhere it should be as well. You never know what might be checking for something without waiting for the postback to finish.

Conner

C

Conner•5/23/24, 5:48 AM

Kk will do

Josiah Bryan

JJosiah Bryan I'll see if I can get permission to share anything. I know I won't be able to sh...

Advany

A

Advany•5/23/24, 9:47 AM

Hi there, where you able to discuss if parts of the docker / controller code can be shared?

Advany

AAdvany Hi there, where you able to discuss if parts of the docker / controller code can...

Josiah Bryan

J

Josiah Bryan•5/23/24, 6:37 PM

I was able to get this code all approved to be shared: https://gist.github.com/josiahbryan/35d7534234f2652648c10e4b13a0730e

in it, you'll find:
- DockerfileDockerfile
  - the file we use to build our k8 image
- browser-service.ymlbrowser-service.yml
  - How we run the docker container in our cluster
- routes.jsroutes.js
  - The routes we expose (via expressexpress, not shown)
- session-controller.jssession-controller.js
  - How we manage sessions
- browser-controller.jsbrowser-controller.js
  - How we load Puppeteer
    Also are a few other utils

mportant note:

We found that we MUST use Puppeteer 17.1.3 for our builds - I think due to compatibility with the wallaby-esmwallaby-esm module, but not entirely sure. Here's the package versions we're using:

    "puppeteer": "^17.1.3",
    "puppeteer-extra": "^3.3.6",
    "puppeteer-extra-plugin-stealth": "^2.11.2",

    "puppeteer": "^17.1.3",
    "puppeteer-extra": "^3.3.6",
    "puppeteer-extra-plugin-stealth": "^2.11.2",

Just FYI - hope some of that helps! Feel free to hit me up with quesitons.

Disclaimer: Not all code is original, some is built on examples found elsewhere online! We just stiched it all together to make it work with our benchwork and cluster and our usecases

Gist

Dockerfile/K8 Setup example for running Puppeteer

Dockerfile/K8 Setup example for running Puppeteer. GitHub Gist: instantly share code, notes, and snippets.

Ian

IIan https://blog.adamcameron.me/2021/01/listening-to-console-log-of-page-loaded.html

Conner

C

Conner•5/23/24, 7:57 PM

Well, found something haha

{
  "logs": [
    "WalkMe: Incompatible browser.",
    "JSHandle@object"
  ]
}

{
  "logs": [
    "WalkMe: Incompatible browser.",
    "JSHandle@object"
  ]
}

Result of page.on("console", (log) => logs.push(log.text()));page.on("console", (log) => logs.push(log.text()));

Conner

C

Conner•5/23/24, 7:58 PM

And when I check the page in chrome it does have a bunch of walkme.comwalkme.com scripts in the head

Conner

CConner And when I check the page in chrome it does have a bunch of `walkme.com` scripts...

Ian

I

IanOP•5/23/24, 8:04 PM

Ian

IIan Click to see attachment

Conner

C

Conner•5/23/24, 8:17 PM

Interesting, Ill see what chrome version cloudflare puppeteer is using then

Conner

C

Conner•5/23/24, 8:23 PM

Sigh - "browserVersion": "Chrome/115.0.5790.184"

Conner

C

Conner•5/23/24, 8:23 PM

So not that

Conner

C

Conner•5/23/24, 8:35 PM

Ah I've done it!

Conner

C

Conner•5/23/24, 8:36 PM

Changed the page to have these after newPage

await page.setCacheEnabled(false);
await page.setViewport({ width: 1920, height: 1080 })
await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36');
await page.setJavaScriptEnabled(true);
await page.evaluateOnNewDocument(() => {
    // Pass the Webdriver Test.
    Object.defineProperty(navigator, 'webdriver', {
        get: () => false,
    });
});

await page.setCacheEnabled(false);
await page.setViewport({ width: 1920, height: 1080 })
await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36');
await page.setJavaScriptEnabled(true);
await page.evaluateOnNewDocument(() => {
    // Pass the Webdriver Test.
    Object.defineProperty(navigator, 'webdriver', {
        get: () => false,
    });
});

Conner

C

Conner•5/23/24, 8:36 PM

Didn't need that on local puppeteer weirdly

Conner

C

Conner•5/23/24, 8:37 PM

@Ian Thanks so much for your help
True hero

<body>
    <form method="post" action="./action.aspx" onkeypress="javascript:return WebForm_FireDefaultButton(event, 'criteria_basic_btn_search')" id="form1">
        <div class="aspNetHidden">
            <input type="hidden" name="__EVENTTARGET" id="__EVENTTARGET" value=""/>
            <input type="hidden" name="__EVENTARGUMENT" id="__EVENTARGUMENT" value=""/>
            <input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="...e"/>
        </div>
        <script type="text/javascript">
            //<![CDATA[
            var theForm = document.forms['form1'];
            if (!theForm) {
                theForm = document.form1;
            }
            function __doPostBack(eventTarget, eventArgument) {
                if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
                    theForm.__EVENTTARGET.value = eventTarget;
                    theForm.__EVENTARGUMENT.value = eventArgument;
                    theForm.submit();
                }
            }
            //]]>
        </script>
        ...

<body>
    <form method="post" action="./action.aspx" onkeypress="javascript:return WebForm_FireDefaultButton(event, 'criteria_basic_btn_search')" id="form1">
        <div class="aspNetHidden">
            <input type="hidden" name="__EVENTTARGET" id="__EVENTTARGET" value=""/>
            <input type="hidden" name="__EVENTARGUMENT" id="__EVENTARGUMENT" value=""/>
            <input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="...e"/>
        </div>
        <script type="text/javascript">
            //<![CDATA[
            var theForm = document.forms['form1'];
            if (!theForm) {
                theForm = document.form1;
            }
            function __doPostBack(eventTarget, eventArgument) {
                if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
                    theForm.__EVENTTARGET.value = eventTarget;
                    theForm.__EVENTARGUMENT.value = eventArgument;
                    theForm.submit();
                }
            }
            //]]>
        </script>
        ...

function __doPostBack(eventTarget, eventArgument) {
    if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
        theForm.__EVENTTARGET.value = eventTarget;
        theForm.__EVENTARGUMENT.value = eventArgument;
        theForm.submit();
    }
}

function __doPostBack(eventTarget, eventArgument) {
    if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
        theForm.__EVENTTARGET.value = eventTarget;
        theForm.__EVENTARGUMENT.value = eventArgument;
        theForm.submit();
    }
}

function __doPostBack(eventTarget, eventArgument) {
    if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
        theForm.__EVENTTARGET.value = eventTarget;
        theForm.__EVENTARGUMENT.value = eventArgument;
        theForm.submit();
    }
}

function __doPostBack(eventTarget, eventArgument) {
    if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
        theForm.__EVENTTARGET.value = eventTarget;
        theForm.__EVENTARGUMENT.value = eventArgument;
        theForm.submit();
    }
}

pageNumber += 1;

await page.waitForFunction(() => typeof window.__doPostBack === 'function', { timeout: 30000, polling: 100 });

await Promise.all([
    page.waitForNavigation({ waitUntil: 'networkidle0' }),
    page.evaluate(`__doPostBack('gridItems','Page$${pageNumber}')`),
]);

pageNumber += 1;

await page.waitForFunction(() => typeof window.__doPostBack === 'function', { timeout: 30000, polling: 100 });

await Promise.all([
    page.waitForNavigation({ waitUntil: 'networkidle0' }),
    page.evaluate(`__doPostBack('gridItems','Page$${pageNumber}')`),
]);

await page.setCacheEnabled(false);
await page.setViewport({ width: 1920, height: 1080 })
await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36');
await page.setJavaScriptEnabled(true);
await page.evaluateOnNewDocument(() => {
    // Pass the Webdriver Test.
    Object.defineProperty(navigator, 'webdriver', {
        get: () => false,
    });
});

await page.setCacheEnabled(false);
await page.setViewport({ width: 1920, height: 1080 })
await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36');
await page.setJavaScriptEnabled(true);
await page.evaluateOnNewDocument(() => {
    // Pass the Webdriver Test.
    Object.defineProperty(navigator, 'webdriver', {
        get: () => false,
    });
});