Cloudflare Developers•2y ago

Hey! if nothing unexpected comes up, late Q1 🙏 If you have feedback or ideas on how this should id

Hey! if nothing unexpected comes up, late Q1

If you have feedback or ideas on how this should ideally work, please do pass them along

PPedro Hey! if nothing unexpected comes up, late Q1 🙏 If you have feedback or ideas o...

jason•1/8/24, 8:48 AM

What I would like is to be able to start a "Durable Browser" and be charged per "Durable Browser" session started. CF would then handle all that behind the scenes stuff to keep the browser open and running (and restarting after crashes, or memory leaks, or whatever) and I don't have to worry about it. And then I can know in my code that I'm only registering 1 or 5 or 10, and then I can manage it accordingly. I'm not sure how that would work from a scalability perspective. But I can't stomach the volatility of either being charged pennies or thousands of dollars because of a memory leak in the browser, which I can't control, crashing and restarting. Or even things like deploying new code or attaching to the log viewer; if I understand right, my durable objects get reset; which starts a new browser session.

jason•1/8/24, 8:49 AM

I guess that's not super helpful. All in all, I don't really care about implementation details. I just need predictability in the pricing.

Jjason What I would like is to be able to start a "Durable Browser" and be charged per ...

PedroOP•1/8/24, 9:05 AM

That's really helpful and aligned we our concerns. Thanks

captain•1/8/24, 7:45 PM

questions about sessions and number of pages and geography. I saw that for $5 a month you get 500 sessions. I was wanting to check multiple search results on certain google searches. 1. Is there a way to define the cloudflare geographic node making the remote request. 2. Is there a limit to how many pages I could view in a single session? 3. Can I block this service from other cloudflare customers using this on my domains? 4. (general puppeteer question..can't find a conclusive answer in google) Is there a good suggestion on how to add a simple timestamp to screen shot with the lowest overhead?

Ccaptain questions about sessions and number of pages and geography. I saw that for $5 a ...

captain•1/8/24, 11:32 PM

(for #4, I injected a new div node with the time stamp before taking a screenshot and that solved my issue!) Now reading back about needing to create and keep open a durable object to maintain a single session if necessary to avoid going over 500 and seeing that it says open as long as there is not a empty 60 seconds. SO if I had had a cront job that checked perhaps a 100 pages with a sleep delay of 3 seconds to allow the content to load before doing a screen shot, and say this cron runs once a day then theoretically it can run 15 minutes giving me the opportuity to run 15 * 60 / 3 = 300 pages in a single cron job max per session

PedroOP•1/9/24, 8:07 AM

Hey @captain, you can indeed keep a session open using durable objects and check multiple search results with the same session. Please find an example here: https://developers.cloudflare.com/browser-rendering/get-started/browser-rendering-with-do/

We are making changed to the product so that in the future you won't have to manage the DO and keepint it alive. We also have plans to add geolocation, so you can choose where the request will be originated, but it is not yet available in current beta. Thanks for sharing your solution for the timestamp

Deploy a Browser Rendering Worker with Durable Objects · Browser Re...

By following this guide, you will create a Worker that uses the Browser Rendering API along with Durable Objects to take screenshots from web pages …

PPedro Hey @captain, you can indeed keep a session open using durable objects and check...

captain•1/9/24, 4:32 PM

thanks pedro for the update that's great to hear geo location and not have to manage a DO. I dont want to maintain a DO as its one more thing to think about (and haven't played with a DO yet...so far I'm rocking the D1 database and R2 storage)

ExSidius•1/10/24, 4:42 PM

Hey guys - I've signed up for the waitlist. Is there anything else I need to do to get access to the beta?

zegevlier•1/10/24, 4:54 PM

There is! Put your account ID in this thread: browser-rendering-apiBrowser Rendering API Beta Access

captain•1/13/24, 5:58 AM

why is puppetteer seeming way faster within the last 15 min?

Jjason Hello. I'm taking a screenshot of websites with the `screenshot()` method, and e...

georgehenderson•1/13/24, 4:55 PM

@jason did you figure out how to make it not take a screenshot of the full page?

georgehenderson•1/13/24, 5:12 PM

I was able to work around this bug by manually clipping the viewport which is a bit annoying but works.

page.screenshot({ clip: { x: 0, y: lastScroll || 0, width: data.innerWidth, height: data.innerHeight } })

page.screenshot({ clip: { x: 0, y: lastScroll || 0, width: data.innerWidth, height: data.innerHeight } })

georgehenderson•1/16/24, 1:26 PM

this does work, but scrollbars appear

usmannk•1/16/24, 2:04 PM

so.. concurrency is very bad when using browser rendering DOs right? If two requests occur simultaneously, one will often overwrite the puppeteer "page" while the other is waiting for page load. What's the recommended way to avoid this? Durable Object input gates are not suitable for this purpose

usmannk•1/16/24, 2:05 PM

is blockConcurrencyWhileblockConcurrencyWhile the way to go?

zegevlier•1/16/24, 2:06 PM

I was looking up the docs link for that haha. Yea, I believe that would work here. Alternatively, there's #queues-beta.

usmannk•1/16/24, 2:07 PM

yeah that makes sense, I was also looking at queues and noticed they have customizable concurrency

zegevlier•1/16/24, 2:08 PM

Although I'm not entirely sure it's needed? Haven't played around with it too much, but would you not be able to get the existing browser, then create a new page, and use that page? I don't think that overwrites it?

usmannk•1/16/24, 2:08 PM

you're right, but eventually you'd have too many tabs open

usmannk•1/16/24, 2:08 PM

tabs == pages

zegevlier•1/16/24, 2:09 PM

Yea that's true, although if you use something like queues you could prevent it from going over the specified concurrency

usmannk•1/16/24, 2:09 PM

yeah, but im not sure how to figure out which page is safe to use

usmannk•1/16/24, 2:09 PM

even if concurrency is limited to, say, 5

zegevlier•1/16/24, 2:10 PM

Why not have each process create a new page, then close it when it's done?

usmannk•1/16/24, 2:10 PM

because what I wouldnt want is to send 5 concurrents reqs: pagecount = 5
send 5 more concurrent reqs: pagecount = 10
etc

Zzegevlier Why not have each process create a new page, then close it when it's done?

usmannk•1/16/24, 2:11 PM

yeah I need to make sure I can close by reference

usmannk•1/16/24, 2:11 PM

e.g. that it's page.close() and not browser.close(4)

usmannk•1/16/24, 2:11 PM

if former, then I think you're right

Zzegevlier Why not have each process create a new page, then close it when it's done?

usmannk•1/16/24, 2:11 PM

yeah ok so I think this works

usmannk•1/16/24, 2:13 PM

I wish there were some slightly more finegrained concurrency primitives, like a cross-worker semaphore

usmannk•1/16/24, 2:13 PM

but it's a bit niche and probably hard to implement

usmannk•1/16/24, 2:13 PM

thanks for the help

Uusmannk I wish there were some slightly more finegrained concurrency primitives, like a ...

usmannk•1/16/24, 2:15 PM

queues kinda do this but im hesitant to add more products

zegevlier•1/16/24, 2:18 PM

Yea that's fair

Uusmannk I wish there were some slightly more finegrained concurrency primitives, like a ...

John Spurlock•1/16/24, 5:52 PM

hmm, semaphore-as-a-service market opportunity

JJohn Spurlock hmm, semaphore-as-a-service market opportunity

DaniFoldi•1/16/24, 5:57 PM

https://mutex.uno built by Greg meowparty

mutex.uno

A free API service for acquiring mutexes.

Patrick•1/18/24, 5:30 PM

Hi everyone, I've not actually got access yet just theory crafting out my usecase... as far as I understand it, in order to have say thousands of requests/pages without immediately bankrupting myself I would need to go the Durable Object route and that basically instantiates a browser that I can re-use but need to keep alive with alarms? I could then queue up my requests to the DO-Browser so that I don't get any issues with puppeteer etc. I've seen further up this thread @Pedro mention that in the future there will be a built-in way to do this? Would this essentially mean that if I had enough traffic to justify it in the future I would be able to keep 1 or more Browser instances running at all times that are being fed requests by a queue, and that way I wouldn't incur the 10c charge per session or require too many concurrent sessions? Thanks!

PPatrick Hi everyone, I've not actually got access yet just theory crafting out my usecas...

PedroOP•1/18/24, 5:36 PM

Hey @Patrick, That's it! You can already keep the session open yourself,, avoiding the creation and charge of new sessions (taking into account the limits), by following the tutorial using durable objects. However, we understand that some customers just want to user Browser Rendering without the overhead of adding code to manage the Durable Objects or even be concerned about less predictable pricing. That's why we are working on an alternative to make it transparent for the user to keep the session open, as you mentioned

PedroOP•1/18/24, 5:37 PM

Also, ask for access on #Browser Rendering API Beta Access and try it out yourself

We're always looking for feedback and suggestions

Patrick•1/18/24, 5:43 PM

Thanks @Pedro! Yeah the future alternative plan sounds perfect, though I'll definitely try the Durable Object way to try it out now. I was thinking that it would be easy enough to set up a way of tracking Opened Browsers within the last month with the Analytics Engine anyway as a failsafe!

Original message was deleted

PedroOP•1/22/24, 10:31 AM

If things go as planned, we should be able to release this by the end of this quarter

Original message was deleted

PedroOP•1/22/24, 5:25 PM

#Browser Rendering API Beta Access please

Original message was deleted

PedroOP•1/24/24, 8:45 AM

Free during beta

But you can find info on the planned price on the announcement blog post: https://blog.cloudflare.com/running-serverless-puppeteer-workers-durable-objects/

jason•1/25/24, 1:50 PM

I'm doing some more testing with this and have run into this error:

Error: Unabled to create new browser: code: 429: message: Too many browsers already runningError: Unabled to create new browser: code: 429: message: Too many browsers already running

Ignoring the typo...

I understand why it's happening. I'm doing development and when my script restarts, the DO gets reset and I lose my reference to the active browser and it starts a new one. Fair enough... but what am I supposed to do now? I can't start new ones, and I can't figure out how to access the running ones to stop them.

jason•1/25/24, 1:58 PM

Actually, this seems like it might not be my fault. Or, at least, there are other problems. According to the docs, the browsers I've opened should have closed after a minute of inactivity anyways.

jason•1/25/24, 1:59 PM

Unless my alarms are still running? I was using wrangler dev --remotewrangler dev --remote to test my code. Could it be the case that exiting out of that didn't shut down my DO alarm which is keeping my browser alive? Hm..

jason•1/26/24, 5:54 AM

This morning, the problem is no longer happening. So whatever happened got cleaned up by itself.

Jjason This morning, the problem is no longer happening. So whatever happened got clean...

PedroOP•1/26/24, 11:46 AM

Thanks for reporting this, we're working on a way to provide you with your open sessions, so you can reuse them if you want to

sty•1/26/24, 1:32 PM

what is the best method to maintain a browser session without it auto closing?
A browser instance gets killed if it does not get any command for 60 seconds, freeing one instance.A browser instance gets killed if it does not get any command for 60 seconds, freeing one instance.

Hey! if nothing unexpected comes up, late Q1 🙏 If you have feedback or ideas on how this should id

Similar Threads