Cloudflare Developers•3y ago

actually this example seems to have lots of mistakes 😅

actually this example seems to have lots of mistakes

Uusmannk actually this example seems to have lots of mistakes 😅

Johannes•11/11/23, 9:16 AM

have you figured out how it should be implemented instead?

JJohannes have you figured out how it should be implemented instead?

usmannkOP•11/11/23, 6:30 PM

at a minimum:

remove the
```
browser.close()
```
```
browser.close()
```
line
check that the browser is connected every fetch (
```
this.browser.isConnected()
```
```
this.browser.isConnected()
```
), and reinstantiate it if not
do not call newPage() every fetch, instead check the browser instance and see if it has the number of pages you need already, from last time
add an await to the setAlarm call

unfortunately it seems we also have to worry about synchronization, if two worker calls grab the browser durable object simultaneously then there is no hope. i think

state.blockConcurrencyWhile()

state.blockConcurrencyWhile()

may be the answer, but I haven't tested it yet

Pedro•11/13/23, 8:57 AM

Hey @uk, this blog post might be a better example of what you're trying to achieve: https://blog.cloudflare.com/running-serverless-puppeteer-workers-durable-objects/

The Cloudflare Blog

Running Serverless Puppeteer with Workers and Durable Objects

We’ve heard from developers that configuring and maintaining their own serverless browser automation systems can be quite painful. The Workers Browser Rendering API solves this

PPedro Hey @_uk_, this blog post might be a better example of what you're trying to ac...

usmannkOP•11/13/23, 3:23 PM

at first glance this is the same code as the linked doc

Johannes•11/14/23, 5:38 AM

Ok, I'm completely confused. From what I see here, and from the lack of responses from Cloudflare, the browser rendering API looks like a dangerous endeavor. It won't scale on high load due to the limitations and it's potentially super expensive

JJohannes Ok, I'm completely confused. From what I see here, and from the lack of response...

Pedro•11/14/23, 8:17 AM

Hey @haushoppe, I believe this was previously addressed, mentioning the use of Durable Object as a way to reuse a browser instance without incurring costs for starting a new session

PPedro Hey @haushoppe, I believe this was previously addressed, mentioning the use of D...

Johannes•11/14/23, 2:35 PM

But @uk has pointed out that the examples are broken, and the discussion was about keeping the instances running via a cronjob - otherwise it gets expensive

Pedro•11/14/23, 2:58 PM

@haushoppe, we'll review the provided example and update it if necessary. However, the key takeaway is that it's possible to reuse the browser instance without additional costs associated with initiating a new session

PPedro @haushoppe, we'll review the provided example and update it if necessary. Howeve...

Johannes•11/14/23, 3:02 PM

Yes, but I have to keep it active within 60 seconds or so. If I somehow fail to update it correctly, then new instances will spawn up and make me poor...

Bram•11/14/23, 6:07 PM

Hi all, I also have a question about the pricing.

Quoted from the blog post:

We’ve spoken to many customers deploying a Puppeteer service on their own infrastructure, on public cloud containers or functions or using managed services. The common theme that we’ve heard is that these services are costly – costly to maintain and expensive to run.

If you create a new browser session every 60 seconds, that means 24 hours * 60 minutes = 1440 sessions per day if you have continuous traffic (at least a request per minute...). If you multiply this by 30 days, that would be 43.200 sessions * 0.10$ = $4320

That seems extremely expensive to me, especially if you only have a couple of requests per minute? I argue that running on own infrastructure would be way cheaper, or am I misunderstanding something here?

BBram Hi all, I also have a question about the pricing. Quoted from the blog post: ...

Bram•11/14/23, 6:08 PM

And this already assumes you are using a durable object to limit the concurrent sessions to #1

Bram•11/14/23, 6:09 PM

I love the convience so far, but am a little worried about the pricing, how definite are those numbers?

James•11/15/23, 2:22 AM

Yeah, the pricing here really doesn’t seem to be competitive, even when compared to more premium solutions such as https://www.browserless.io.

It would be really nice to hear some clarifications and thoughts around the pricing as it exists today, for how it was determined.

Browserless - #1 Web Automation & Headless Browser Automation Tool

Try Browserless, one of the best web automation tools for free. Implement web scraping, PDF generation & headless browser automation easily.

Unsmart•11/15/23, 2:34 AM

Seems browserless allows a lot more concurrency but the charge per session can be more than cloudflare if you keep the session alive with durable objects

You could in theory do 15,000 sessions ($25 on browserless) with a single session on cloudflare ($0.10).

Unsmart•11/15/23, 2:34 AM

And the more requests you do keeping a session alive the cheaper it becomes compared to browserless

Unsmart•11/15/23, 2:35 AM

Main difference there is you sort of have to manage the load of the durable objects yourself a bit

James•11/15/23, 2:36 AM

In my opinion, any product that requires a user to scaffold something like a DO to manage things themselves to ensure their pricing doesn’t explode is a really bad design for serverless and the rest of Cloudflare’s product stack.

zegevlier•11/15/23, 6:40 AM

This is my largest issue with the current pricing. For it to be at all competitive, you need to manage it state fully, which is so opposed to the rest of the product stack

UUnsmart Seems browserless allows a lot more concurrency but the charge per session can b...

Bram•11/15/23, 6:49 AM

How so? I thought a session can live a maximum of 60 seconds as a limit?

Bram•11/15/23, 6:50 AM

Ah wait, you have to keep it warm

zegevlier•11/15/23, 6:50 AM

They get killed after 60 seconds of inactivity

Bram•11/15/23, 6:50 AM

A browser instance gets killed if it does not get any command for 60 seconds, freeing one instance.

Zzegevlier They get killed after 60 seconds of inactivity

Bram•11/15/23, 6:52 AM

Misunderstood that, so you can keep it alive for months on end and only pay for the durable object?

BBram Misunderstood that, so you can keep it alive for months on end and only pay for ...

Mackenly•11/15/23, 6:53 AM

Maybe, or maybe not it seems. It depends if the DO gets reset or runs out of memory. That question is really what we've been asking since the pricing was announced. So, maybe, but probably not in real world production (at least not for months).

UUnsmart Main difference there is you sort of have to manage the load of the durable obje...

Bram•11/15/23, 7:02 AM

Which imo is weird for a cloud product, it should be cost optimised behind the scenes and have competitive per usage pricing imo, if you have a special use case that needs full control over the browser instance lifetimes, they could expose an additional API for lower level control?

BBram Which imo is weird for a cloud product, it should be cost optimised behind the s...

Pedro•11/15/23, 11:27 AM

Thank you for your feedback. While our current beta has specific limits and procedures, we're open to evolving them based on feedback and common use cases

Bram•11/15/23, 8:43 PM

Hi, let's say I want to set the html programitcally with

setContent

setContent

and that html runs some javascript that wants to access a large local json file, can puppeteer somehow access the local filestystem of the worker? Or how would this work?

BBram Hi, let's say I want to set the html programitcally with `setContent` and that h...

Bram•11/16/23, 10:30 AM

I ended up just embedding the large json directly into a script tag, not sure if it is the most efficient way, but it works

Ssc You can't access the local file system. You could save the file in R2 (they can ...

Bram•11/16/23, 11:23 AM

yea, in fact the json blob is inside r2 already, but it has to stay private, was thinking about using a signed url to generate temporary access, but was wondering if there was a better way. For now directly embedding it into the html (like so:

<script> const json = JSON.parse(${rawJson};  ...

<script> const json = JSON.parse(${rawJson};  ...

) and then use

page.setContents

page.setContents

works, but not sure if this is a performant way to do it.

jo32•11/18/23, 3:24 PM

How do I disable CORS when launching a browser?

GGustavo I think it's clear already the current pricing is at least very confusing and sh...

Pedro•11/20/23, 9:43 AM

Hey @gustavopch, we're still in beta and are actively learning from all the use-cases and great feedback we're getting. This is really helpful for us as we fine-tune our pricing to better suit your needs. Thanks for bearing with us!

Eevo4ce Hi Pedro, great to hear you’re receiving great feedback from the community. In...

Pedro•11/21/23, 11:00 AM

Hey @evo4ce, sure, as an update I can tell you that we're considering alternative approaches so that the pricing concerns are addressed, namely, keeping sessions open without the need for the users to manage Durable Objects/state

PPedro Hey @evo4ce, sure, as an update I can tell you that we're considering alternativ...

Vikash•11/21/23, 12:02 PM

At-least add some reports so that we can calculate how many sessions, requests we are consuming in Beta for better planning the and costs.

Ssc fyi we updated the code in the Durable Object example https://developers.cloudfl...

Vikash•11/21/23, 4:46 PM

Still the page is not closed/re-used.

So we will end up creating too many stalled pages after few requests.

Add ‘await page.close()’

Ssc fyi we updated the code in the Durable Object example https://developers.cloudfl...

Tsopic•11/25/23, 8:37 AM

Would love to test out this Browser rendering API.

Currently using Google cloud for puppeteer lambdas for email preview renders etc, but dis too slow for meh. Joined the beta access waitlist

TTsopic Would love to test out this Browser rendering API. Currently using Google clou...

Vero•11/26/23, 1:55 PM

To get access tomorrow, please drop your account ID in the thread: Browser Rendering API Beta Access

barretlee•11/30/23, 1:40 AM

Is there a way to quickly get browser rendering access in worker?

barretlee•11/30/23, 1:41 AM

Thanks you @Vero

, got it.

daodao97.•11/30/23, 8:02 AM

Hi, could I get access for 14438683d1c010b14269b6c993f8c048 Thank you in advance!

Ddaodao97.Hi, could I get access for 14438683d1c010b14269b6c993f8c048 Thank you in advance...

zegevlier•11/30/23, 10:56 AM

Post your account ID in this thread: Browser Rendering API Beta Access

Paige•12/4/23, 3:16 PM

It seems like the @\cloudflare/puppeteer npm package is not working entirely as intended as the regular puppeteer does - for example, taking a screenshot of a youtube page, 1st image is using puppeteer on my pc (same result when using a vps I have), and the 2nd image is puppeteer hosted using the browser rendering api on cloudflare using @\cloudflare/puppeteer npm package

For reference, I did use almost exactly the same code, the puppeteer one:

    const browser = await pup.launch({
        headless: `new`,
        defaultViewport: { width: 1920, height: 1080 },
    })

    const getPageSnapshot = (url, fullPage) => new Promise(async (resolve, reject) => { // this returns a promise resolving to a buffer
        const page = await browser.newPage().catch(reject)
        await page.goto(url).catch(reject)

        resolve(await page.screenshot({
            fullPage
        }).then(async (r) => { await page.close(); return r }).catch(reject))
    })

    const browser = await pup.launch({
        headless: `new`,
        defaultViewport: { width: 1920, height: 1080 },
    })

    const getPageSnapshot = (url, fullPage) => new Promise(async (resolve, reject) => { // this returns a promise resolving to a buffer
        const page = await browser.newPage().catch(reject)
        await page.goto(url).catch(reject)

        resolve(await page.screenshot({
            fullPage
        }).then(async (r) => { await page.close(); return r }).catch(reject))
    })

And the cloudflare one:

    async handle(
        request: Request,
        env: any,
        context: any,
        data: Record<string, any>
    ) : Promise<Response> {
                // some url handling
        
        const browser = await puppeteer.launch(env.MYBROWSER);
        const page = await browser.newPage();
        await page.setViewport({ width: 1920, height: 1080 })
        await page.goto(url);
        const screenshot = (await page.screenshot({
            fullPage: false,
        })) as Buffer;
        await browser.close();

        return new Response(screenshot, {
            headers: {
                'Content-Type': `image/jpeg`,
            },
        });
    }

    async handle(
        request: Request,
        env: any,
        context: any,
        data: Record<string, any>
    ) : Promise<Response> {
                // some url handling
        
        const browser = await puppeteer.launch(env.MYBROWSER);
        const page = await browser.newPage();
        await page.setViewport({ width: 1920, height: 1080 })
        await page.goto(url);
        const screenshot = (await page.screenshot({
            fullPage: false,
        })) as Buffer;
        await browser.close();

        return new Response(screenshot, {
            headers: {
                'Content-Type': `image/jpeg`,
            },
        });
    }

PPaige It seems like the @\cloudflare/puppeteer npm package is not working entirely as ...

Unsmart•12/4/23, 3:31 PM

try adding a waitUntil "load" on the page goto

UUnsmart try adding a waitUntil "load" on the page goto

Paige•12/4/23, 3:39 PM

using

        await page.goto(url, {
            waitUntil: `load`
        });

        await page.goto(url, {
            waitUntil: `load`
        });

gives an identical result

Jürgen B•12/5/23, 9:52 AM

I'm getting the following error "Exceeded Resources" in my logs for my scheduled execs, however it seems my script works just great, any ideas?

const browser = await pup.launch({ headless: `new`, defaultViewport: { width: 1920, height: 1080 }, }) const getPageSnapshot = (url, fullPage) => new Promise(async (resolve, reject) => { // this returns a promise resolving to a buffer const page = await browser.newPage().catch(reject) await page.goto(url).catch(reject) resolve(await page.screenshot({ fullPage }).then(async (r) => { await page.close(); return r }).catch(reject)) })

async handle( request: Request, env: any, context: any, data: Record<string, any> ) : Promise<Response> { // some url handling const browser = await puppeteer.launch(env.MYBROWSER); const page = await browser.newPage(); await page.setViewport({ width: 1920, height: 1080 }) await page.goto(url); const screenshot = (await page.screenshot({ fullPage: false, })) as Buffer; await browser.close(); return new Response(screenshot, { headers: { 'Content-Type': `image/jpeg`, }, }); }

actually this example seems to have lots of mistakes 😅

Similar Threads

Similar Threads

Similar Threads