@Pedro Another question... one of the things that excites me most about this project is the hope of

@Pedro Another question... one of the things that excites me most about this project is the hope of scalability. When I've run puppeteer myself on my servers, the challenge that I've always run into it trying to handle scaling it up. Trying to throttle requests to not start too many processes and run the server out of memory, etc. Is one of the goals of this project actually to be able to solve that?

Is this eventually going to work like other Workers do, where I don't need to care about how many requests I throw at it? Or am I going to have to implement all of the throttling code anyways - but now with the limitation that I can't see what's going on inside the server?

Jjason @Pedro Another question... one of the things that excites me most about this pro...

Pedro•2/1/24, 2:10 PM

Great question! We'll be providing a simple way to reuse sessions and keeping them open, so you can focus on the task at hand. We'll still be setting default limits, mostly to prevent abuse and malicious behaviour, but those can be raised according to you use case

PPedro Great question! We'll be providing a simple way to reuse sessions and keeping th...

jasonOP•2/1/24, 2:13 PM

Thanks. Another way to ask the question. Let's say that I have 1000 urls to crawl. Am I going to need to do the work to manually distribute those requests over multiple sessions, throttle them to not send too many at a time, etc? With a normal worker that doesn't use the browser rendering, I would just POST all 1000 requests as fast as possible to my worker and not worry about it. But I suspect with the browser rendering, I wouldn't be able to do that.

Pedro•2/1/24, 2:19 PM

You can choose to reuse the same session (until expired) to crawls the URLs sequentially or you can choose to create concurrent sessions. It's basically up to you. We're providing an easier way to reuse the sessions, but the "puppeteer" behaviour is still up to you to define, so it's flexible enough to accommodate most use cases. (You can already reuse the same session for multiple URLs using Durable Objects as per the tutorial, we're making it easier)

jasonOP•2/1/24, 2:20 PM

Are you saying that a single session can handle one request at a time? So if I wanted to do 1000 requests in parallel, I would need to start 1000 separate sessions?

Jjason Are you saying that a single session can handle one request at a time? So if I w...

Pedro•2/1/24, 2:25 PM

You can use the same session for multiple tasks like crawling multiple URLs. But if you want to do things in parallel, you just create new concurrent sessions. It's like running a Browser and doing the tasks or Multiple Browsers at the same time and performing multiple task in each one at the same time. Hope I was clear enough

jasonOP•2/1/24, 2:31 PM

Yep, I think I understand.

And I think it means that it's not going to have the worker-level scalability that I hoped it would. At least not without costing a fortune. Thanks for the information!

Jjason Yep, I think I understand. 👍 And I think it means that it's not going to have t...

Pedro•2/1/24, 2:53 PM

Would love to learn more about your use case. Please feel free to ping me and let's make sure we have an offer that also suits more specific and demanding use cases such as yours

SSHAH hello, how do i get off the waiting list for this? i've been on it for ages

jasonOP•2/2/24, 5:21 AM

Did you post in #Browser Rendering API Beta Access ? It’s usually within 24 hours.

kalepail•2/2/24, 7:05 PM

Does this API work in dev? I'm not seeing any indication that it shouldn't but when calling a method that works in prod I'm getting

Could not start browser instance. Error: TypeError: Cannot read properties of undefined (reading 'fetch')

Could not start browser instance. Error: TypeError: Cannot read properties of undefined (reading 'fetch')

chientrm.com•2/3/24, 10:33 AM

uhm, does this api work for Pages?

Kkalepail Does this API work in dev? I'm not seeing any indication that it shouldn't but w...

jasonOP•2/3/24, 1:53 PM

You need to use

wrangler dev --remote

wrangler dev --remote

Yacine Hmito•2/9/24, 2:39 PM

@sc Hello! Did you have the chance to run the Puppeeteer fork locally? Thank you.

Ssc Hi @Yacine Just did and can confirm. Will have to look into it.

Yacine Hmito•2/10/24, 8:50 AM

Thank you!

Original message was deleted

Pedro•2/12/24, 8:37 AM

Hey! You can use DOs, regardless of using one session or multiple concurrent sessions. Basically, using DOs allows you to reuse the same session by keeping it open longer, thus if you don't have the need to crawl or execute commands in parallel, one session should be enough. If you have the need to execute commands in parallel, then you can open new "concurrent" sessions

Erisa•2/14/24, 12:54 PM

You can request access in browser-rendering-apiBrowser Rendering API Beta Access

Erisa•2/14/24, 12:54 PM

And you can see in the chat history that it doesnt take too long

Erisa•2/14/24, 12:56 PM

I don't see a message from you in that thread

MyBuddhaMercy•2/22/24, 7:01 AM

Does cloudflare puppeteer support setting proxy?

Prabesh•2/22/24, 7:43 AM

facing the error

Dynamic require of "node:stream" is not supported

Dynamic require of "node:stream" is not supported

when using the legacy syntax with addEventListener.

MyBuddhaMercy•2/24/24, 11:59 AM

How to use head mode when debugging locally ?

f12i•2/26/24, 9:44 PM

I just got access to the browser rendering API and I can't seem to get it to work. The response from puppeteer.launch(...) is null and I see a "Unabled to create new browser: code: 429: message: Too many browsers already running". I am running just this one instance for testing, is the system currently overloaded?

f12i•2/26/24, 9:56 PM

Well, it just started working. It was some transient issue.

Wang•2/27/24, 2:14 AM

Hi, Cloudflare staff

Wang•2/27/24, 2:16 AM

When I use local Node.js Puppeteer to access the website https://help.openai.com/en/articles/7102672-how-can-i-access-gpt-4, all content can be fully loaded. However, when using Cloudflare Browser Render to access it, not all content can be loaded, only some JavaScript scripts. What could be the reason for this?

Wang•2/27/24, 2:17 AM

This the local Node.js Puppeteer code:

const browser = await puppeteer.launch({headless: false});
const page = await browser.newPage();
const url = "https://help.openai.com/en/articles/7102672-how-can-i-access-gpt-4";
await page.goto(url, { waitUntil: 'networkidle0' }); // Ensure page is fully loaded

const content = await page.content();
console.log(content);

await browser.close();

const browser = await puppeteer.launch({headless: false});
const page = await browser.newPage();
const url = "https://help.openai.com/en/articles/7102672-how-can-i-access-gpt-4";
await page.goto(url, { waitUntil: 'networkidle0' }); // Ensure page is fully loaded

const content = await page.content();
console.log(content);

await browser.close();

Wang•2/27/24, 2:19 AM

This the worker code

const browser = await puppeteer.launch(env.MYBROWSER);
const page = await browser.newPage();
await page.goto(url, { waitUntil: 'networkidle0' }); // Ensure page is fully loaded

const content = await page.content();
console.log(content);
await browser.close();

const browser = await puppeteer.launch(env.MYBROWSER);
const page = await browser.newPage();
await page.goto(url, { waitUntil: 'networkidle0' }); // Ensure page is fully loaded

const content = await page.content();
console.log(content);
await browser.close();

Ssc @Yacine In order to _build_ it locally, you'll need to run node v18 (that error ...

Yacine Hmito•2/27/24, 9:18 AM

Thank you. I'll try again with node 18 (although I think this is what I used the first time, but no harm in checking)

gauravwagmi•2/28/24, 10:41 AM

Hey folks, how can I avoid issues like this, for Google eg, whiel trying to screenshot, using the browser rendering API

gauravwagmi•2/28/24, 10:42 AM

andycyz•2/28/24, 12:34 PM

Hi, how can I get access?

Aandycyz Hi, how can I get access?

Vero•2/28/24, 12:45 PM

Hey, drop your account ID in the thread browser-rendering-apiBrowser Rendering API Beta Access

Ggauravwagmi Hey folks, how can I avoid issues like this, for Google eg, whiel trying to scre...

jasonOP•2/29/24, 5:21 AM

It's unlikely that you are going to be able to, or at least find support here for how to. You have to convince reCaptcha, that you aren't a bot... but since you are, you might have a hard time doing that.

gauravwagmi•2/29/24, 10:08 AM

I believe there is a puppeteer stealth plugin, which I'd love an integration wtih. Not an ace programmer, so please bear with me.

gauravwagmi•2/29/24, 10:08 AM

https://www.npmjs.com/package/puppeteer-extra-plugin-stealth

npm

puppeteer-extra-plugin-stealth

Stealth mode: Applies various techniques to make detection of headless puppeteer harder.. Latest version: 2.11.2, last published: a year ago. Start using puppeteer-extra-plugin-stealth in your project by running

npm i puppeteer-extra-plugin-stealth

npm i puppeteer-extra-plugin-stealth

. There are 523 other projects in the npm registry using puppeteer-extra-plugin-stealth.

MyBuddhaMercy•3/5/24, 3:34 PM

For the same browser instance, if users in different countries request and execute page actions, are the exit IP addresses through the page the same?

MyBuddhaMercy•3/5/24, 3:37 PM

What determines the IP of the browser? Is it possible that the IP of the same browser instance in different contexts or pages is different?

MMyBuddhaMercy For the same browser instance, if users in different countries request and execu...

jasonOP•3/5/24, 4:02 PM

Once you start a session, the browser remains in that location until it's closed. Your cannot move the browser to different locations. I'd imagine that the browser is opened in a location close to the request that it was opened; though I don't know that for sure.

gauravwagmi•3/6/24, 2:48 PM

Hey folks, is there a way we can increase these limits? I'd love to have ~5 a minute. https://developers.cloudflare.com/browser-rendering/platform/limits/ Maybe @Vero

can help?

Cloudflare Docs

Limits · Browser Rendering docs

The Workers Browser Rendering API is currently in beta. Add yourself to the waitlist to reserve your spot, and as we expand availability, we will …

Isaac McFadyen•3/6/24, 7:04 PM

?pings

Flare•3/6/24, 7:04 PM

Please do not ping community members for non-moderation reasons. Doing so will not solve your issue faster and will make people less likely to want to help you.

Ggauravwagmi Hey folks, is there a way we can increase these limits? I'd love to have ~5 a mi...

Vero•3/7/24, 5:00 PM

Hey, not sure if those limits can be increased but I've asked internally

Ggauravwagmi Hey folks, is there a way we can increase these limits? I'd love to have ~5 a mi...

Vero•3/7/24, 5:30 PM

and they just told me that Browser Rendering limits can't be increased currently

Ilmerainen•3/8/24, 5:58 PM

Hey!
Is there a way to set userDataDir in puppeteer.launch()?

const browser = await puppeteer.launch({headless: false}); const page = await browser.newPage(); const url = "https://help.openai.com/en/articles/7102672-how-can-i-access-gpt-4"; await page.goto(url, { waitUntil: 'networkidle0' }); // Ensure page is fully loaded const content = await page.content(); console.log(content); await browser.close();

const browser = await puppeteer.launch(env.MYBROWSER); const page = await browser.newPage(); await page.goto(url, { waitUntil: 'networkidle0' }); // Ensure page is fully loaded const content = await page.content(); console.log(content); await browser.close();

@Pedro Another question... one of the things that excites me most about this project is the hope of

Similar Threads