I wish there were some slightly more finegrained concurrency primitives, like a cross-worker semapho
I wish there were some slightly more finegrained concurrency primitives, like a cross-worker semaphore
Ubut it's a bit niche and probably hard to implement 
Uthanks for the help
Uqueues kinda do this but im hesitant to add more products
ZYea that's fair
Jhmm, semaphore-as-a-service market opportunity
D
A free API service for acquiring mutexes.
PHi everyone, I've not actually got access yet just theory crafting out my usecase... as far as I understand it, in order to have say thousands of requests/pages without immediately bankrupting myself I would need to go the Durable Object route and that basically instantiates a browser that I can re-use but need to keep alive with alarms? I could then queue up my requests to the DO-Browser so that I don't get any issues with puppeteer etc. I've seen further up this thread @Pedro mention that in the future there will be a built-in way to do this? Would this essentially mean that if I had enough traffic to justify it in the future I would be able to keep 1 or more Browser instances running at all times that are being fed requests by a queue, and that way I wouldn't incur the 10c charge per session or require too many concurrent sessions? Thanks!
PHey @Patrick, That's it! You can already keep the session open yourself,, avoiding the creation and charge of new sessions (taking into account the limits), by following the tutorial using durable objects. However, we understand that some customers just want to user Browser Rendering without the overhead of adding code to manage the Durable Objects or even be concerned about less predictable pricing. That's why we are working on an alternative to make it transparent for the user to keep the session open, as you mentioned
PAlso, ask for access on #Browser Rendering API Beta Access and try it out yourself 
We're always looking for feedback and suggestions
We're always looking for feedback and suggestionsPThanks @Pedro! Yeah the future alternative plan sounds perfect, though I'll definitely try the Durable Object way to try it out now. I was thinking that it would be easy enough to set up a way of tracking Opened Browsers within the last month with the Analytics Engine anyway as a failsafe!
PIf things go as planned, we should be able to release this by the end of this quarter 
P#Browser Rendering API Beta Access please 
PFree during beta But you can find info on the planned price on the announcement blog post: https://blog.cloudflare.com/running-serverless-puppeteer-workers-durable-objects/
But you can find info on the planned price on the announcement blog post: https://blog.cloudflare.com/running-serverless-puppeteer-workers-durable-objects/
JI'm doing some more testing with this and have run into this error:
Ignoring the typo...
Error: Unabled to create new browser: code: 429: message: Too many browsers already runningIgnoring the typo...
- I understand why it's happening. I'm doing development and when my script restarts, the DO gets reset and I lose my reference to the active browser and it starts a new one. Fair enough... but what am I supposed to do now? I can't start new ones, and I can't figure out how to access the running ones to stop them.
JActually, this seems like it might not be my fault. Or, at least, there are other problems. According to the docs, the browsers I've opened should have closed after a minute of inactivity anyways.
JUnless my alarms are still running? I was using
wrangler dev --remote to test my code. Could it be the case that exiting out of that didn't shut down my DO alarm which is keeping my browser alive? Hm..JThis morning, the problem is no longer happening. So whatever happened got cleaned up by itself.
PThanks for reporting this, we're working on a way to provide you with your open sessions, so you can reuse them if you want to
Swhat is the best method to maintain a browser session without it auto closing?
A browser instance gets killed if it does not get any command for 60 seconds, freeing one instance.Sahh i got it nvm:
YI'm trying to hack on @Cloudflare/puppeteer locally. I can't get anything to install this is driving me crazy.
YI pull the repo, run
npm install with node 18 and it fails with "Cannot use import statement outside a module", even though, according to the GitHub Actions workflows, that is supposed to work fine.YI tried converting the offending files to
.mts but that only moves the problem somwhere else.YI tried with the upstream puppeteer on the commit that was forked and the issue is the same 
DHmm, if the package is esm-only (which I haven't looked into), could it be that you have to be in a module for it to work? Wild guess, all this cjs vs esm nonsense is driving me crazy too, but could you try adding
"type": "module" to your package.json?
YThe problem is that only the TypeScript files are in ESM. All the rest is in CJS.
YSorry to ping you @celso but I didn't manage to run @Cloudflare/puppeteer locally because of the problem I ran into above.
(Context: I am attempting to reproduce the service that manages chromium instances so that I can use the Browser Rendering API locally with workerd)
(Context: I am attempting to reproduce the service that manages chromium instances so that I can use the Browser Rendering API locally with workerd)
JI'm not sure where that error is coming from, but it may be coming from within your browser context. Buffer isn't defined in the browser's javascript. Maybe the
extractData method is trying to use it?JCan you provide the stacktrace of the error?
JAdd
console.log(e.stack) after the "page on response error"
CI'm not too familiar with puppeter's APIs, but node support with workers is like this:
node_compat=true in your wrangler.toml polyfills some of the nodejs functionality like buffer, ex
nodejs_compat flag would only expose it via
What version of cloudflare/puppeter are you using? The latest should work with nodejs_compat flag, but before 0.0.4 release it needed node_compat
node_compat=true in your wrangler.toml polyfills some of the nodejs functionality like buffer, ex
import { Buffer } from 'buffer' would just worknodejs_compat flag would only expose it via
node:buffer, have to specifically use it, ex import { Buffer } from 'node:buffer'.What version of cloudflare/puppeter are you using? The latest should work with nodejs_compat flag, but before 0.0.4 release it needed node_compat
DHi - Whats the best way to bypass when websites block us from taking a screenshot?
J1) Request the wishes of the website and don't take the screenshot
2) How exactly are they blocking you?
2) How exactly are they blocking you?
DCloudflare antibody firewall
JI'm not sure you are going to get many tips about how to circumvent Cloudflare's Firewall on the Cloudflare Discord...
YThere's no new code. I just pull the repo and did
npm install. I checked that I had a compatible version of node set, but it still didn't work.JCan someone explain how this works in CF's distributed network? When a browser is started, is it only in a single location? If I wanted browsers in multiple locations, how would I handle that?
PHey jason! Currently it is started on a single location, we'll be adding support for Geolocation, so that for example the content rendered might change according to the location where connection is initiated
JThanks as always @Pedro
J@Pedro Another question... one of the things that excites me most about this project is the hope of scalability. When I've run puppeteer myself on my servers, the challenge that I've always run into it trying to handle scaling it up. Trying to throttle requests to not start too many processes and run the server out of memory, etc. Is one of the goals of this project actually to be able to solve that?
Is this eventually going to work like other Workers do, where I don't need to care about how many requests I throw at it? Or am I going to have to implement all of the throttling code anyways - but now with the limitation that I can't see what's going on inside the server?
Is this eventually going to work like other Workers do, where I don't need to care about how many requests I throw at it? Or am I going to have to implement all of the throttling code anyways - but now with the limitation that I can't see what's going on inside the server?
PGreat question! We'll be providing a simple way to reuse sessions and keeping them open, so you can focus on the task at hand. We'll still be setting default limits, mostly to prevent abuse and malicious behaviour, but those can be raised according to you use case
JThanks. Another way to ask the question. Let's say that I have 1000 urls to crawl. Am I going to need to do the work to manually distribute those requests over multiple sessions, throttle them to not send too many at a time, etc? With a normal worker that doesn't use the browser rendering, I would just POST all 1000 requests as fast as possible to my worker and not worry about it. But I suspect with the browser rendering, I wouldn't be able to do that.
PYou can choose to reuse the same session (until expired) to crawls the URLs sequentially or you can choose to create concurrent sessions. It's basically up to you. We're providing an easier way to reuse the sessions, but the "puppeteer" behaviour is still up to you to define, so it's flexible enough to accommodate most use cases. (You can already reuse the same session for multiple URLs using Durable Objects as per the tutorial, we're making it easier)
JAre you saying that a single session can handle one request at a time? So if I wanted to do 1000 requests in parallel, I would need to start 1000 separate sessions?
