Why not have each process create a new page, then close it when it's done?

usmannk•1/16/24, 2:10 PM

because what I wouldnt want is to send 5 concurrents reqs: pagecount = 5
send 5 more concurrent reqs: pagecount = 10
etc

Zzegevlier Why not have each process create a new page, then close it when it's done?

usmannk•1/16/24, 2:11 PM

yeah I need to make sure I can close by reference

usmannk•1/16/24, 2:11 PM

e.g. that it's page.close() and not browser.close(4)

usmannk•1/16/24, 2:11 PM

if former, then I think you're right

Zzegevlier Why not have each process create a new page, then close it when it's done?

usmannk•1/16/24, 2:11 PM

yeah ok so I think this works

usmannk•1/16/24, 2:13 PM

I wish there were some slightly more finegrained concurrency primitives, like a cross-worker semaphore

usmannk•1/16/24, 2:13 PM

but it's a bit niche and probably hard to implement

usmannk•1/16/24, 2:13 PM

thanks for the help

Uusmannk I wish there were some slightly more finegrained concurrency primitives, like a ...

usmannk•1/16/24, 2:15 PM

queues kinda do this but im hesitant to add more products

zegevlierOP•1/16/24, 2:18 PM

Yea that's fair

Uusmannk I wish there were some slightly more finegrained concurrency primitives, like a ...

John Spurlock•1/16/24, 5:52 PM

hmm, semaphore-as-a-service market opportunity

JJohn Spurlock hmm, semaphore-as-a-service market opportunity

DaniFoldi•1/16/24, 5:57 PM

https://mutex.uno built by Greg meowparty

mutex.uno

A free API service for acquiring mutexes.

Patrick•1/18/24, 5:30 PM

Hi everyone, I've not actually got access yet just theory crafting out my usecase... as far as I understand it, in order to have say thousands of requests/pages without immediately bankrupting myself I would need to go the Durable Object route and that basically instantiates a browser that I can re-use but need to keep alive with alarms? I could then queue up my requests to the DO-Browser so that I don't get any issues with puppeteer etc. I've seen further up this thread @Pedro mention that in the future there will be a built-in way to do this? Would this essentially mean that if I had enough traffic to justify it in the future I would be able to keep 1 or more Browser instances running at all times that are being fed requests by a queue, and that way I wouldn't incur the 10c charge per session or require too many concurrent sessions? Thanks!

PPatrick Hi everyone, I've not actually got access yet just theory crafting out my usecas...

Pedro•1/18/24, 5:36 PM

Hey @Patrick, That's it! You can already keep the session open yourself,, avoiding the creation and charge of new sessions (taking into account the limits), by following the tutorial using durable objects. However, we understand that some customers just want to user Browser Rendering without the overhead of adding code to manage the Durable Objects or even be concerned about less predictable pricing. That's why we are working on an alternative to make it transparent for the user to keep the session open, as you mentioned

Pedro•1/18/24, 5:37 PM

Also, ask for access on #Browser Rendering API Beta Access and try it out yourself

We're always looking for feedback and suggestions

Patrick•1/18/24, 5:43 PM

Thanks @Pedro! Yeah the future alternative plan sounds perfect, though I'll definitely try the Durable Object way to try it out now. I was thinking that it would be easy enough to set up a way of tracking Opened Browsers within the last month with the Analytics Engine anyway as a failsafe!

Original message was deleted

Pedro•1/22/24, 10:31 AM

If things go as planned, we should be able to release this by the end of this quarter

Original message was deleted

Pedro•1/22/24, 5:25 PM

#Browser Rendering API Beta Access please

Original message was deleted

Pedro•1/24/24, 8:45 AM

Free during beta

But you can find info on the planned price on the announcement blog post: https://blog.cloudflare.com/running-serverless-puppeteer-workers-durable-objects/

jason•1/25/24, 1:50 PM

I'm doing some more testing with this and have run into this error:

Error: Unabled to create new browser: code: 429: message: Too many browsers already running

Error: Unabled to create new browser: code: 429: message: Too many browsers already running

Ignoring the typo...

I understand why it's happening. I'm doing development and when my script restarts, the DO gets reset and I lose my reference to the active browser and it starts a new one. Fair enough... but what am I supposed to do now? I can't start new ones, and I can't figure out how to access the running ones to stop them.

jason•1/25/24, 1:58 PM

Actually, this seems like it might not be my fault. Or, at least, there are other problems. According to the docs, the browsers I've opened should have closed after a minute of inactivity anyways.

jason•1/25/24, 1:59 PM

Unless my alarms are still running? I was using

wrangler dev --remote

wrangler dev --remote

to test my code. Could it be the case that exiting out of that didn't shut down my DO alarm which is keeping my browser alive? Hm..

jason•1/26/24, 5:54 AM

This morning, the problem is no longer happening. So whatever happened got cleaned up by itself.

Jjason This morning, the problem is no longer happening. So whatever happened got clean...

Pedro•1/26/24, 11:46 AM

Thanks for reporting this, we're working on a way to provide you with your open sessions, so you can reuse them if you want to

sty•1/26/24, 1:32 PM

what is the best method to maintain a browser session without it auto closing?

A browser instance gets killed if it does not get any command for 60 seconds, freeing one instance.

A browser instance gets killed if it does not get any command for 60 seconds, freeing one instance.

sty•1/26/24, 1:35 PM

ahh i got it nvm:

    async alarm() {
        this.keptAliveInSeconds += 10;

        // Extend browser DO life
        if (this.keptAliveInSeconds < KEEP_BROWSER_ALIVE_IN_SECONDS) {

    async alarm() {
        this.keptAliveInSeconds += 10;

        // Extend browser DO life
        if (this.keptAliveInSeconds < KEEP_BROWSER_ALIVE_IN_SECONDS) {

Yacine Hmito•1/27/24, 8:47 AM

I'm trying to hack on @Cloudflare/puppeteer locally. I can't get anything to install this is driving me crazy.

Yacine Hmito•1/27/24, 8:49 AM

I pull the repo, run

npm install

npm install

with node 18 and it fails with "Cannot use import statement outside a module", even though, according to the GitHub Actions workflows, that is supposed to work fine.

Yacine Hmito•1/27/24, 8:49 AM

I tried converting the offending files to

.mts

.mts

but that only moves the problem somwhere else.

Yacine Hmito•1/27/24, 8:51 AM

I tried with the upstream puppeteer on the commit that was forked and the issue is the same verysad

YYacine Hmito I pull the repo, run `npm install` with node 18 and it fails with "Cannot use im...

DaniFoldi•1/27/24, 1:46 PM

Hmm, if the package is esm-only (which I haven't looked into), could it be that you have to be in a module for it to work? Wild guess, all this cjs vs esm nonsense is driving me crazy too, but could you try adding

"type": "module"

"type": "module"

to your package.json?

DDaniFoldi Hmm, if the package is esm-only (which I haven't looked into), could it be that ...

Yacine Hmito•1/29/24, 11:18 AM

The problem is that only the TypeScript files are in ESM. All the rest is in CJS.

Yacine Hmito•1/29/24, 11:35 AM

Sorry to ping you @celso but I didn't manage to run @Cloudflare/puppeteer locally because of the problem I ran into above.
(Context: I am attempting to reproduce the service that manages chromium instances so that I can use the Browser Rendering API locally with workerd)

LLivog Hey! I'm testing out cloudflare/puppeteer api So I m running this code: ``` pag...

jason•1/29/24, 2:24 PM

I'm not sure where that error is coming from, but it may be coming from within your browser context. Buffer isn't defined in the browser's javascript. Maybe the

extractData

extractData

method is trying to use it?

LLivog Thanks for the replay, the extractData is simplifying a object to only get the d...

jason•1/29/24, 2:27 PM

Can you provide the stacktrace of the error?

LLivog Never done that, got any link of instructions on how to do that?

jason•1/29/24, 2:30 PM

Add

console.log(e.stack)

console.log(e.stack)

after the "page on response error"

LLivog Hey! I'm testing out cloudflare/puppeteer api So I m running this code: ``` pag...

Chaika•1/29/24, 2:32 PM

I'm not too familiar with puppeter's APIs, but node support with workers is like this:
node_compat=true in your wrangler.toml polyfills some of the nodejs functionality like buffer, ex

import { Buffer } from 'buffer'

import { Buffer } from 'buffer'

would just work
nodejs_compat flag would only expose it via

node:buffer

node:buffer

, have to specifically use it, ex

import { Buffer } from 'node:buffer'

import { Buffer } from 'node:buffer'

.

What version of cloudflare/puppeter are you using? The latest should work with nodejs_compat flag, but before 0.0.4 release it needed node_compat

Denny•1/29/24, 11:49 PM

Hi - Whats the best way to bypass when websites block us from taking a screenshot?

DDenny Hi - Whats the best way to bypass when websites block us from taking a screensho...

jason•1/30/24, 4:01 AM

1) Request the wishes of the website and don't take the screenshot
2) How exactly are they blocking you?

Jjason 1) Request the wishes of the website and don't take the screenshot 2) How exactl...

Denny•1/30/24, 7:11 AM

Cloudflare antibody firewall

DDenny Cloudflare antibody firewall

jason•1/30/24, 8:52 AM

I'm not sure you are going to get many tips about how to circumvent Cloudflare's Firewall on the Cloudflare Discord...

Ssc Hey @Yacine Any chance you put a minimal example in a gist we could look at?

Yacine Hmito•1/30/24, 12:46 PM

There's no new code. I just pull the repo and did

npm install

npm install

. I checked that I had a compatible version of node set, but it still didn't work.

jason•2/1/24, 6:47 AM

Can someone explain how this works in CF's distributed network? When a browser is started, is it only in a single location? If I wanted browsers in multiple locations, how would I handle that?

Jjason Can someone explain how this works in CF's distributed network? When a browser i...

Pedro•2/1/24, 9:54 AM

Hey jason! Currently it is started on a single location, we'll be adding support for Geolocation, so that for example the content rendered might change according to the location where connection is initiated

Why not have each process create a new page, then close it when it's done?

Similar Threads

Similar Threads

Similar Threads