My other next thought is maybe inspect the page network requests? Making sure they line up with what

I

Ian•5/23/24, 5:31 AM

https://blog.adamcameron.me/2021/01/listening-to-console-log-of-page-loaded.html

Listening to the console log of a page loaded with Puppeteer

Adam Cameron works out how to listen for a Puppeteer page's log activities.

CConner My other next thought is maybe inspect the page network requests? Making sure th...

I

Ian•5/23/24, 5:32 AM

yeah, though, you already have the JS you need right there baked into the page.

I

Ian•5/23/24, 5:32 AM

My current guess is something is crashing and blocking that function from being created.

I

Ian•5/23/24, 5:33 AM

@Conner quick question, are you literally calling

__doCallBack

__doCallBack

__doCallBack

__doCallBack ? Or clicking the submit button?

IIan @Conner quick question, are you literally calling `__doCallBack` ? Or clicking t...

C

ConnerOP•5/23/24, 5:34 AM

I've tried both, neither worked

C

ConnerOP•5/23/24, 5:34 AM

I also tried running the code inside __doCallBack

C

ConnerOP•5/23/24, 5:35 AM

function __doPostBack(eventTarget, eventArgument) {
    if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
        theForm.__EVENTTARGET.value = eventTarget;
        theForm.__EVENTARGUMENT.value = eventArgument;
        theForm.submit();
    }
}

function __doPostBack(eventTarget, eventArgument) {
    if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
        theForm.__EVENTTARGET.value = eventTarget;
        theForm.__EVENTARGUMENT.value = eventArgument;
        theForm.submit();
    }
}

function __doPostBack(eventTarget, eventArgument) {
    if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
        theForm.__EVENTTARGET.value = eventTarget;
        theForm.__EVENTARGUMENT.value = eventArgument;
        theForm.submit();
    }
}

function __doPostBack(eventTarget, eventArgument) {
    if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
        theForm.__EVENTTARGET.value = eventTarget;
        theForm.__EVENTARGUMENT.value = eventArgument;
        theForm.submit();
    }
}

When I tried running that code inside, I got something like cannot set property value of undefined

C

ConnerOP•5/23/24, 5:35 AM

So theForm.__EVENTTARGET is undefined? Which again is weird because the whole page visually loads in fine

I

Ian•5/23/24, 5:36 AM

Right because

theForm

theForm

theForm

theForm isn't defined in there, it's defined just outside by

var theForm = document.forms['form1'];

var theForm = document.forms['form1'];

var theForm = document.forms['form1'];

var theForm = document.forms['form1'];

CConner The script for this is in the body if that makes any difference ``` <body> ...

C

ConnerOP•5/23/24, 5:36 AM

It's a global variable but yeah I also tried querySelector #form1

I

Ian•5/23/24, 5:39 AM

so what's supposed to happen is when a button on the aspx page is clicked it will call

__doPostBack

__doPostBack

__doPostBack

__doPostBack and fill in the eventTarget and any eventArgs. These then get filled in as those hidden

__EVENTTARGET

__EVENTTARGET

__EVENTTARGET

__EVENTTARGET and

__EVENTARGUMENT

__EVENTARGUMENT

__EVENTARGUMENT

__EVENTARGUMENT input fields defined above. Those get submitted to the server and that's how it knows what button you pressed so it can run the appropriate event handler code on the server, then send you back a new page render.

I

Ian•5/23/24, 5:40 AM

so you should definitely be clicking directly on a button if you can help it to make sure that gets filled out correctly.

C

ConnerOP•5/23/24, 5:43 AM

In some places 100%

But some I know what I need to fill in so I'm just straight up calling evaluate, for example

pageNumber += 1;

await page.waitForFunction(() => typeof window.__doPostBack === 'function', { timeout: 30000, polling: 100 });

await Promise.all([
    page.waitForNavigation({ waitUntil: 'networkidle0' }),
    page.evaluate(`__doPostBack('gridItems','Page$${pageNumber}')`),
]);

pageNumber += 1;

await page.waitForFunction(() => typeof window.__doPostBack === 'function', { timeout: 30000, polling: 100 });

await Promise.all([
    page.waitForNavigation({ waitUntil: 'networkidle0' }),
    page.evaluate(`__doPostBack('gridItems','Page$${pageNumber}')`),
]);

pageNumber += 1;

await page.waitForFunction(() => typeof window.__doPostBack === 'function', { timeout: 30000, polling: 100 });

await Promise.all([
    page.waitForNavigation({ waitUntil: 'networkidle0' }),
    page.evaluate(`__doPostBack('gridItems','Page$${pageNumber}')`),
]);

pageNumber += 1;

await page.waitForFunction(() => typeof window.__doPostBack === 'function', { timeout: 30000, polling: 100 });

await Promise.all([
    page.waitForNavigation({ waitUntil: 'networkidle0' }),
    page.evaluate(`__doPostBack('gridItems','Page$${pageNumber}')`),
]);

C

ConnerOP•5/23/24, 5:44 AM

I'll check console and lyk, I think you're write it might be failing somehow and logging to console

I

Ian•5/23/24, 5:46 AM

double check you've got

await

await

everywhere it should be as well. You never know what might be checking for something without waiting for the postback to finish.

C

ConnerOP•5/23/24, 5:48 AM

Kk will do

JJosiah Bryan I'll see if I can get permission to share anything. I know I won't be able to sh...

A

Advany•5/23/24, 9:47 AM

Hi there, where you able to discuss if parts of the docker / controller code can be shared?

AAdvany Hi there, where you able to discuss if parts of the docker / controller code can...

J

Josiah Bryan•5/23/24, 6:37 PM

I was able to get this code all approved to be shared: https://gist.github.com/josiahbryan/35d7534234f2652648c10e4b13a0730e

in it, you'll find:
- ```
Dockerfile
```
```
Dockerfile
```
  - the file we use to build our k8 image
- ```
browser-service.yml
```
  browser-service.yml
```
browser-service.yml
```
  browser-service.yml
  - How we run the docker container in our cluster
- ```
routes.js
```
  routes.js
```
routes.js
```
  routes.js
  - The routes we expose (via
    express
    express
    express
    express
    , not shown)
- ```
session-controller.js
```
  session-controller.js
```
session-controller.js
```
  session-controller.js
  - How we manage sessions
- ```
browser-controller.js
```
  browser-controller.js
```
browser-controller.js
```
  browser-controller.js
  - How we load Puppeteer
    Also are a few other utils

mportant note:

We found that we MUST use Puppeteer 17.1.3 for our builds - I think due to compatibility with the

wallaby-esm

wallaby-esm

wallaby-esm

wallaby-esm module, but not entirely sure. Here's the package versions we're using:

    "puppeteer": "^17.1.3",
    "puppeteer-extra": "^3.3.6",
    "puppeteer-extra-plugin-stealth": "^2.11.2",

    "puppeteer": "^17.1.3",
    "puppeteer-extra": "^3.3.6",
    "puppeteer-extra-plugin-stealth": "^2.11.2",

    "puppeteer": "^17.1.3",
    "puppeteer-extra": "^3.3.6",
    "puppeteer-extra-plugin-stealth": "^2.11.2",

    "puppeteer": "^17.1.3",
    "puppeteer-extra": "^3.3.6",
    "puppeteer-extra-plugin-stealth": "^2.11.2",

Just FYI - hope some of that helps! Feel free to hit me up with quesitons.

Disclaimer: Not all code is original, some is built on examples found elsewhere online! We just stiched it all together to make it work with our benchwork and cluster and our usecases

Gist

Dockerfile/K8 Setup example for running Puppeteer

Dockerfile/K8 Setup example for running Puppeteer. GitHub Gist: instantly share code, notes, and snippets.

IIan https://blog.adamcameron.me/2021/01/listening-to-console-log-of-page-loaded.html

C

ConnerOP•5/23/24, 7:57 PM

Well, found something haha

{
  "logs": [
    "WalkMe: Incompatible browser.",
    "JSHandle@object"
  ]
}

{
  "logs": [
    "WalkMe: Incompatible browser.",
    "JSHandle@object"
  ]
}

{
  "logs": [
    "WalkMe: Incompatible browser.",
    "JSHandle@object"
  ]
}

{
  "logs": [
    "WalkMe: Incompatible browser.",
    "JSHandle@object"
  ]
}

Result of

page.on("console", (log) => logs.push(log.text()));

page.on("console", (log) => logs.push(log.text()));

page.on("console", (log) => logs.push(log.text()));

page.on("console", (log) => logs.push(log.text()));

C

ConnerOP•5/23/24, 7:58 PM

And when I check the page in chrome it does have a bunch of

walkme.com

walkme.com

walkme.com

walkme.com scripts in the head

CConner And when I check the page in chrome it does have a bunch of `walkme.com` scripts...

I

Ian•5/23/24, 8:04 PM

IIan Click to see attachment

C

ConnerOP•5/23/24, 8:17 PM

Interesting, Ill see what chrome version cloudflare puppeteer is using then

C

ConnerOP•5/23/24, 8:23 PM

Sigh - "browserVersion": "Chrome/115.0.5790.184"

C

ConnerOP•5/23/24, 8:23 PM

So not that

C

ConnerOP•5/23/24, 8:35 PM

Ah I've done it!

C

ConnerOP•5/23/24, 8:36 PM

Changed the page to have these after newPage

await page.setCacheEnabled(false);
await page.setViewport({ width: 1920, height: 1080 })
await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36');
await page.setJavaScriptEnabled(true);
await page.evaluateOnNewDocument(() => {
    // Pass the Webdriver Test.
    Object.defineProperty(navigator, 'webdriver', {
        get: () => false,
    });
});

await page.setCacheEnabled(false);
await page.setViewport({ width: 1920, height: 1080 })
await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36');
await page.setJavaScriptEnabled(true);
await page.evaluateOnNewDocument(() => {
    // Pass the Webdriver Test.
    Object.defineProperty(navigator, 'webdriver', {
        get: () => false,
    });
});

await page.setCacheEnabled(false);
await page.setViewport({ width: 1920, height: 1080 })
await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36');
await page.setJavaScriptEnabled(true);
await page.evaluateOnNewDocument(() => {
    // Pass the Webdriver Test.
    Object.defineProperty(navigator, 'webdriver', {
        get: () => false,
    });
});

await page.setCacheEnabled(false);
await page.setViewport({ width: 1920, height: 1080 })
await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36');
await page.setJavaScriptEnabled(true);
await page.evaluateOnNewDocument(() => {
    // Pass the Webdriver Test.
    Object.defineProperty(navigator, 'webdriver', {
        get: () => false,
    });
});

C

ConnerOP•5/23/24, 8:36 PM

Didn't need that on local puppeteer weirdly

C

ConnerOP•5/23/24, 8:37 PM

@Ian Thanks so much for your help
True hero

CConner Didn't need that on local puppeteer weirdly

I

Ian•5/23/24, 8:39 PM

Most likely it's doing something like using your local chrome which has slightly different settings or whatever.

I

Ian•5/23/24, 8:40 PM

If you really want to do a clean room test, run it locally inside Docker.

C

ConnerOP•5/23/24, 8:44 PM

Ah it works, so good to go haha

C

ConnerOP•5/23/24, 8:45 PM

Walkme probably didn't like some headers that the headless browser was sending

C

ConnerOP•5/23/24, 8:45 PM

Until I set those things

E

eric•5/27/24, 10:30 AM

Hey! Is the render api still in private beta? I've signed up on the page, but the workers docs doesn't mention ıt anywhere. Keep getting the

workers.api.error.no_access_to_browser_worker

workers.api.error.no_access_to_browser_worker

workers.api.error.no_access_to_browser_worker

workers.api.error.no_access_to_browser_worker, so I'd assume so!

Wwingeva Can cloudflare@puppeteer be used together with puppeteer-extra?

D

Dan Claroni•5/28/24, 10:05 PM

did you ever get an answer to this?

JJosiah Bryan I was able to get this code all approved to be shared: https://gist.github.com/j...

A

Advany•5/29/24, 7:32 AM

You are awesome!!!!

S

sam.parry•5/29/24, 8:49 AM

Super excited to get browser rendering live for our use case!

When did it come out of invite only?

I've done some testing locally, and all is working awesome. I have a question regarding fonts though, when I generate a screenshot it is generating with a rather ugly system font (perhaps windows / linux default?)

Is there any way we can configure the default font? Ideally to use something neurtral and clean like Helvetica or San Fransisco?

Google Fonts seem to work well, but wondering if there is a built in approach?

D

Dan Claroni•5/29/24, 6:07 PM

Does anyone have any experience getting past security on API requests using puppeteer on workers? I can easily get past it with this code using AWS Lambda, but I cant seem to get around this using the cloudflare workers implementation:

code:

const [page] = await browser.pages();

// Hijack browser navigation and instead send requests.
await page.setRequestInterception(true);
page.once('request', async (interceptedRequest) => {
    const headers = {};
    // eslint-disable-next-line @typescript-eslint/no-explicit-any
    request.headers.forEach((v, k) => ((headers as any)[k] = v));
    interceptedRequest.continue({
        method: request.method,
        postData: requestBody,
        headers,
    });

    await page.setRequestInterception(false);
});

// Trigger the requests
const response = await page.goto(request.url);

const [page] = await browser.pages();

// Hijack browser navigation and instead send requests.
await page.setRequestInterception(true);
page.once('request', async (interceptedRequest) => {
    const headers = {};
    // eslint-disable-next-line @typescript-eslint/no-explicit-any
    request.headers.forEach((v, k) => ((headers as any)[k] = v));
    interceptedRequest.continue({
        method: request.method,
        postData: requestBody,
        headers,
    });

    await page.setRequestInterception(false);
});

// Trigger the requests
const response = await page.goto(request.url);

const [page] = await browser.pages();

// Hijack browser navigation and instead send requests.
await page.setRequestInterception(true);
page.once('request', async (interceptedRequest) => {
    const headers = {};
    // eslint-disable-next-line @typescript-eslint/no-explicit-any
    request.headers.forEach((v, k) => ((headers as any)[k] = v));
    interceptedRequest.continue({
        method: request.method,
        postData: requestBody,
        headers,
    });

    await page.setRequestInterception(false);
});

// Trigger the requests
const response = await page.goto(request.url);

const [page] = await browser.pages();

// Hijack browser navigation and instead send requests.
await page.setRequestInterception(true);
page.once('request', async (interceptedRequest) => {
    const headers = {};
    // eslint-disable-next-line @typescript-eslint/no-explicit-any
    request.headers.forEach((v, k) => ((headers as any)[k] = v));
    interceptedRequest.continue({
        method: request.method,
        postData: requestBody,
        headers,
    });

    await page.setRequestInterception(false);
});

// Trigger the requests
const response = await page.goto(request.url);

I keep getting 403's with the site giving me a generic

Sorry, you have been blocked

Sorry, you have been blocked

Sorry, you have been blocked

Sorry, you have been blocked error page

T

TTTT•5/30/24, 12:32 AM

what's the pricing for browser-rendering-api, it's really vague

TTTTT what's the pricing for browser-rendering-api, it's really vague

C

Chaika•5/30/24, 2:07 AM

Currently, no pricing, but Workers Paid only (and limited to two browsers concurrent/per minute)
They're working on getting pricing set

CChaika Currently, no pricing, but Workers Paid only (and limited to two browsers concur...

T

TTTT•5/30/24, 2:29 AM

thank you, when do we expect the new pricing to come alive? I just don't want to be surprised with a bill

C

Chaika•5/30/24, 2:30 AM

I don't know of any exact date but should be plenty of emails and notification beforehand

T

TTTT•5/30/24, 2:36 AM

sounds good. another question, the browser session has a cf physical location right? so if the session was created on the edge in cf-sfo, then when try to use the same session on a worker in dublin, then, the speed will be slow, because the sfo session is trying to access resources in dublin.

T

TTTT•5/30/24, 2:36 AM

is my understanding correct?

TTTTT sounds good. another question, the browser session has a cf physical location ...

C

Chaika•5/30/24, 3:06 AM

The idea of reusing a session is to avoid the cost of spinning up a new session, so it moving locations wouldn't make sense, would need a new session

CChaika The idea of reusing a session is to avoid the cost of spinning up a new session,...

T

TTTT•5/30/24, 3:28 AM

thank you, makes sense

P

Paradox_77•6/5/24, 6:53 AM

is the beta still going on or is it over?

P

Paradox_77•6/5/24, 6:53 AM

im looking to get access to the browser rendering api

My other next thought is maybe inspect the page network requests? Making sure they line up with what

Similar Threads

Similar Threads

Similar Threads