Hey there when using the Browser Rendering API, I ran into a cloudflare challenge on a website I wan

Hey there when using the Browser Rendering API, I ran into a cloudflare challenge on a website I wanted to access lol. Is there anyway to wait/bypass that? Or is this a goner? Thanks!

DDanTheGoodman Does puppeteer stealth work? (ping plz)

hellocryptokrOP•11/8/23, 7:53 AM

Hey Dan did stealth work out for you?

Hhellocryptokr Hey Dan did stealth work out for you?

DanTheGoodman•11/8/23, 12:24 PM

I haven't tried it, I don't think anyone answered my question on pricing so I didn't try it

Hhellocryptokr Hey there when using the Browser Rendering API, I ran into a cloudflare challeng...

DanTheGoodman•11/8/23, 12:24 PM

I think that's the point, they don't want bots

DDanTheGoodman I haven't tried it, I don't think anyone answered my question on pricing so I di...

hellocryptokrOP•11/8/23, 12:25 PM

That's a bummer... will probably have to work a way around then. Thanks for answering bro!

Vikash•11/8/23, 12:59 PM

Is this supposed to work on Pages function?

Jjason Hello, I'm trying to understand the proposed pricing structure and how to optimi...

Pedro•11/8/23, 5:38 PM

Hey @jasonlfunk, currently, on the free plan, each account can initiate up to 100 browser sessions per month.
For the Paid plan (included in Workers), each account can initiate up to 500 sessions/month with the capability to run 10 concurrent sessions. Additional sessions on this plan are billed at $0.10 each, and additional concurrent sessions costs $1.00 each.
Each account can start up to two new browser sessions every minute. Note that if a browser instance remains idle, with no commands issued for 60 seconds, it will be terminated, thus making room for a new available instance.
Durable Object enables the creation of a "continuous browser session". This approach allows for the reuse of a browser instance without incurring costs for starting a new session, though charges for the Durable Object apply

PPedro Hey @jasonlfunk, currently, on the free plan, each account can initiate up to 10...

Unsmart•11/8/23, 5:41 PM

Based on this message I assume you work at CF. As an fyi you can run the

/cfrole

/cfrole

command to get the cloudflare employee role

PPedro Hey @jasonlfunk, currently, on the free plan, each account can initiate up to 10...

jason•11/8/23, 5:48 PM

I understand. I think the outstanding questions are this:
1) Is it a problem to keep a browser session open indefinitely? (by using alarms/cron to keep it active).
2) How many pages can a single browser session reasonably have open at a time? (If my app is processing 100 requests per second, can they all be sent to a single session? Should they be split up over multiple sessions?)

Jjason I understand. I think the outstanding questions are this: 1) Is it a problem to ...

Pedro•11/8/23, 5:59 PM

We are currently in beta and things might change, but 1) As shown on the previously mentioned blog post, it's acceptable to keep the session active. 2) I'll need to verify and follow up regarding your second question, but it seems the limitations will likely be tied to memory capacity, which can vary with each use-case

PPedro We are currently in beta and things might change, but 1) As shown on the previou...

jason•11/8/23, 6:03 PM

Great, thanks for the information.

Zzegevlier You can absolutely do that: Keep in mind that the current limits for the beta on...

Vikash•11/9/23, 12:02 PM

Does this means, it cost me $3,880 per month for single URL screenshot every minute?

For example, if I have a worker capturing a website screenshot on every minute scheduled CRON.

(43,800 minutes/session * $0.10 per session) - 500 free session = $3,880

Assuming a new session will be created each time the scheduled function starts.

async scheduled(
    event: ScheduledEvent,
    env: Env,
    ctx: ExecutionContext
  ): Promise<void> {
    const urls = [
      "https://agenty.com",
    ];
    const browser = await puppeteer.launch(env.MYBROWSER);
    await screenshot(
      browser,
      urls,
      env
    );
    await browser.close();
  },

async scheduled(
    event: ScheduledEvent,
    env: Env,
    ctx: ExecutionContext
  ): Promise<void> {
    const urls = [
      "https://agenty.com",
    ];
    const browser = await puppeteer.launch(env.MYBROWSER);
    await screenshot(
      browser,
      urls,
      env
    );
    await browser.close();
  },

It's going to be super expensive

that way. What are other suggestion?

Pedro•11/9/23, 12:17 PM

Hey @Vikash [Agenty], consider using Durable Object as a way to reuse the open sessions. Please take a look at this blog post, as the example seems quite similar to your use-case: https://blog.cloudflare.com/running-serverless-puppeteer-workers-durable-objects/

The Cloudflare Blog

Running Serverless Puppeteer with Workers and Durable Objects

We’ve heard from developers that configuring and maintaining their own serverless browser automation systems can be quite painful. The Workers Browser Rendering API solves this

PPedro Hey @Vikash [Agenty], consider using Durable Object as a way to reuse the open s...

Vikash•11/9/23, 12:31 PM

If I use durable object and keep setting the alarm. Does that mean my session will never expire? Because I am using CRON so it will keep warming up every minute

VVikash If I use durable object and keep setting the alarm. Does that mean my session wi...

Vikash•11/9/23, 12:32 PM

Also, any way to track cost/sessions in beta?

VVikash If I use durable object and keep setting the alarm. Does that mean my session wi...

Pedro•11/9/23, 1:37 PM

Mostly, note that the Durable Object might be terminated in some instances, such as if it is idle for 30 seconds, if it hits memory limits or when we put out new releases

VVikash Also, any way to track cost/sessions in beta?

Pedro•11/9/23, 1:39 PM

Not something we offer at the moment, but we are exploring this possibility

DevGuy•11/10/23, 3:02 PM

Hello, I would like to request access to the browser rendering API. Thanks!

DDevGuy Hello, I would like to request access to the browser rendering API. Thanks!

zegevlier•11/10/23, 3:10 PM

Put your account ID in browser-rendering-apiBrowser Rendering API Beta Access

Zzegevlier Put your account ID in https://discord.com/channels/595317990191398933/111267505...

DevGuy•11/10/23, 3:11 PM

Thanks!

Johannes•11/10/23, 11:29 PM

Wow, I joined this Discord to learn more about the Browser Rendering API. And I'm shocked about the pricing!

For 5 USD i can execute a simple docker container, that might does not scale, but that wouldn't introduce a private bankruptcy...

usmannk•11/11/23, 12:29 AM

https://developers.cloudflare.com/browser-rendering/get-started/browser-rendering-with-do/#6-

Deploy a Browser Rendering Worker with Durable Objects · Browser Re...

By following this guide, you will create a Worker that uses the Browser Rendering API along with Durable Objects to take screenshots from web pages …

usmannk•11/11/23, 12:30 AM

is the example code above correct? The browser instance is unconditionally closed in fetch(), which makes the durable object useless for the next call

usmannk•11/11/23, 12:30 AM

@Pedro

usmannk•11/11/23, 12:39 AM

actually this example seems to have lots of mistakes

Uusmannk actually this example seems to have lots of mistakes 😅

Johannes•11/11/23, 9:16 AM

have you figured out how it should be implemented instead?

JJohannes have you figured out how it should be implemented instead?

usmannk•11/11/23, 6:30 PM

at a minimum:

remove the
```
browser.close()
```
```
browser.close()
```
line
check that the browser is connected every fetch (
```
this.browser.isConnected()
```
```
this.browser.isConnected()
```
), and reinstantiate it if not
do not call newPage() every fetch, instead check the browser instance and see if it has the number of pages you need already, from last time
add an await to the setAlarm call

unfortunately it seems we also have to worry about synchronization, if two worker calls grab the browser durable object simultaneously then there is no hope. i think

state.blockConcurrencyWhile()

state.blockConcurrencyWhile()

may be the answer, but I haven't tested it yet

Pedro•11/13/23, 8:57 AM

Hey @uk, this blog post might be a better example of what you're trying to achieve: https://blog.cloudflare.com/running-serverless-puppeteer-workers-durable-objects/

The Cloudflare Blog

Running Serverless Puppeteer with Workers and Durable Objects

We’ve heard from developers that configuring and maintaining their own serverless browser automation systems can be quite painful. The Workers Browser Rendering API solves this

PPedro Hey @_uk_, this blog post might be a better example of what you're trying to ac...

usmannk•11/13/23, 3:23 PM

at first glance this is the same code as the linked doc

Johannes•11/14/23, 5:38 AM

Ok, I'm completely confused. From what I see here, and from the lack of responses from Cloudflare, the browser rendering API looks like a dangerous endeavor. It won't scale on high load due to the limitations and it's potentially super expensive

JJohannes Ok, I'm completely confused. From what I see here, and from the lack of response...

Pedro•11/14/23, 8:17 AM

Hey @haushoppe, I believe this was previously addressed, mentioning the use of Durable Object as a way to reuse a browser instance without incurring costs for starting a new session

PPedro Hey @haushoppe, I believe this was previously addressed, mentioning the use of D...

Johannes•11/14/23, 2:35 PM

But @uk has pointed out that the examples are broken, and the discussion was about keeping the instances running via a cronjob - otherwise it gets expensive

Pedro•11/14/23, 2:58 PM

@haushoppe, we'll review the provided example and update it if necessary. However, the key takeaway is that it's possible to reuse the browser instance without additional costs associated with initiating a new session

PPedro @haushoppe, we'll review the provided example and update it if necessary. Howeve...

Johannes•11/14/23, 3:02 PM

Yes, but I have to keep it active within 60 seconds or so. If I somehow fail to update it correctly, then new instances will spawn up and make me poor...

Bram•11/14/23, 6:07 PM

Hi all, I also have a question about the pricing.

Quoted from the blog post:

We’ve spoken to many customers deploying a Puppeteer service on their own infrastructure, on public cloud containers or functions or using managed services. The common theme that we’ve heard is that these services are costly – costly to maintain and expensive to run.

If you create a new browser session every 60 seconds, that means 24 hours * 60 minutes = 1440 sessions per day if you have continuous traffic (at least a request per minute...). If you multiply this by 30 days, that would be 43.200 sessions * 0.10$ = $4320

That seems extremely expensive to me, especially if you only have a couple of requests per minute? I argue that running on own infrastructure would be way cheaper, or am I misunderstanding something here?

BBram Hi all, I also have a question about the pricing. Quoted from the blog post: ...

Bram•11/14/23, 6:08 PM

And this already assumes you are using a durable object to limit the concurrent sessions to #1

Bram•11/14/23, 6:09 PM

I love the convience so far, but am a little worried about the pricing, how definite are those numbers?

James•11/15/23, 2:22 AM

Yeah, the pricing here really doesn’t seem to be competitive, even when compared to more premium solutions such as https://www.browserless.io.

It would be really nice to hear some clarifications and thoughts around the pricing as it exists today, for how it was determined.

Browserless - #1 Web Automation & Headless Browser Automation Tool

Try Browserless, one of the best web automation tools for free. Implement web scraping, PDF generation & headless browser automation easily.

Unsmart•11/15/23, 2:34 AM

Seems browserless allows a lot more concurrency but the charge per session can be more than cloudflare if you keep the session alive with durable objects

You could in theory do 15,000 sessions ($25 on browserless) with a single session on cloudflare ($0.10).

Unsmart•11/15/23, 2:34 AM

And the more requests you do keeping a session alive the cheaper it becomes compared to browserless

Unsmart•11/15/23, 2:35 AM

Main difference there is you sort of have to manage the load of the durable objects yourself a bit

James•11/15/23, 2:36 AM

In my opinion, any product that requires a user to scaffold something like a DO to manage things themselves to ensure their pricing doesn’t explode is a really bad design for serverless and the rest of Cloudflare’s product stack.

zegevlier•11/15/23, 6:40 AM

This is my largest issue with the current pricing. For it to be at all competitive, you need to manage it state fully, which is so opposed to the rest of the product stack

UUnsmart Seems browserless allows a lot more concurrency but the charge per session can b...

Bram•11/15/23, 6:49 AM

How so? I thought a session can live a maximum of 60 seconds as a limit?

Bram•11/15/23, 6:50 AM

Ah wait, you have to keep it warm

zegevlier•11/15/23, 6:50 AM

They get killed after 60 seconds of inactivity

Bram•11/15/23, 6:50 AM

A browser instance gets killed if it does not get any command for 60 seconds, freeing one instance.

Zzegevlier They get killed after 60 seconds of inactivity

Bram•11/15/23, 6:52 AM

Misunderstood that, so you can keep it alive for months on end and only pay for the durable object?

BBram Misunderstood that, so you can keep it alive for months on end and only pay for ...

Mackenly•11/15/23, 6:53 AM

Maybe, or maybe not it seems. It depends if the DO gets reset or runs out of memory. That question is really what we've been asking since the pricing was announced. So, maybe, but probably not in real world production (at least not for months).

UUnsmart Main difference there is you sort of have to manage the load of the durable obje...

Bram•11/15/23, 7:02 AM

Which imo is weird for a cloud product, it should be cost optimised behind the scenes and have competitive per usage pricing imo, if you have a special use case that needs full control over the browser instance lifetimes, they could expose an additional API for lower level control?

BBram Which imo is weird for a cloud product, it should be cost optimised behind the s...

Pedro•11/15/23, 11:27 AM

Thank you for your feedback. While our current beta has specific limits and procedures, we're open to evolving them based on feedback and common use cases

async scheduled( event: ScheduledEvent, env: Env, ctx: ExecutionContext ): Promise<void> { const urls = [ "https://agenty.com", ]; const browser = await puppeteer.launch(env.MYBROWSER); await screenshot( browser, urls, env ); await browser.close(); },

Hey there when using the Browser Rendering API, I ran into a cloudflare challenge on a website I wan

Similar Threads

Similar Threads

Similar Threads