Cloudflare Developers•2y ago

hmm, semaphore-as-a-service market opportunity

JJohn Spurlock hmm, semaphore-as-a-service market opportunity

DaniFoldi•1/16/24, 5:57 PM

https://mutex.uno built by Greg meowparty

mutex.uno

A free API service for acquiring mutexes.

Patrick•1/18/24, 5:30 PM

Hi everyone, I've not actually got access yet just theory crafting out my usecase... as far as I understand it, in order to have say thousands of requests/pages without immediately bankrupting myself I would need to go the Durable Object route and that basically instantiates a browser that I can re-use but need to keep alive with alarms? I could then queue up my requests to the DO-Browser so that I don't get any issues with puppeteer etc. I've seen further up this thread @Pedro mention that in the future there will be a built-in way to do this? Would this essentially mean that if I had enough traffic to justify it in the future I would be able to keep 1 or more Browser instances running at all times that are being fed requests by a queue, and that way I wouldn't incur the 10c charge per session or require too many concurrent sessions? Thanks!

PPatrick Hi everyone, I've not actually got access yet just theory crafting out my usecas...

Pedro•1/18/24, 5:36 PM

Hey @Patrick, That's it! You can already keep the session open yourself,, avoiding the creation and charge of new sessions (taking into account the limits), by following the tutorial using durable objects. However, we understand that some customers just want to user Browser Rendering without the overhead of adding code to manage the Durable Objects or even be concerned about less predictable pricing. That's why we are working on an alternative to make it transparent for the user to keep the session open, as you mentioned

Pedro•1/18/24, 5:37 PM

Also, ask for access on #Browser Rendering API Beta Access and try it out yourself

We're always looking for feedback and suggestions

Patrick•1/18/24, 5:43 PM

Thanks @Pedro! Yeah the future alternative plan sounds perfect, though I'll definitely try the Durable Object way to try it out now. I was thinking that it would be easy enough to set up a way of tracking Opened Browsers within the last month with the Analytics Engine anyway as a failsafe!

Original message was deleted

Pedro•1/22/24, 10:31 AM

If things go as planned, we should be able to release this by the end of this quarter

Original message was deleted

Pedro•1/22/24, 5:25 PM

#Browser Rendering API Beta Access please

Original message was deleted

Pedro•1/24/24, 8:45 AM

Free during beta

But you can find info on the planned price on the announcement blog post: https://blog.cloudflare.com/running-serverless-puppeteer-workers-durable-objects/

jason•1/25/24, 1:50 PM

I'm doing some more testing with this and have run into this error:

Error: Unabled to create new browser: code: 429: message: Too many browsers already running

Error: Unabled to create new browser: code: 429: message: Too many browsers already running

Ignoring the typo...

I understand why it's happening. I'm doing development and when my script restarts, the DO gets reset and I lose my reference to the active browser and it starts a new one. Fair enough... but what am I supposed to do now? I can't start new ones, and I can't figure out how to access the running ones to stop them.

jason•1/25/24, 1:58 PM

Actually, this seems like it might not be my fault. Or, at least, there are other problems. According to the docs, the browsers I've opened should have closed after a minute of inactivity anyways.

jason•1/25/24, 1:59 PM

Unless my alarms are still running? I was using

wrangler dev --remote

wrangler dev --remote

to test my code. Could it be the case that exiting out of that didn't shut down my DO alarm which is keeping my browser alive? Hm..

jason•1/26/24, 5:54 AM

This morning, the problem is no longer happening. So whatever happened got cleaned up by itself.

Jjason This morning, the problem is no longer happening. So whatever happened got clean...

Pedro•1/26/24, 11:46 AM

Thanks for reporting this, we're working on a way to provide you with your open sessions, so you can reuse them if you want to

sty•1/26/24, 1:32 PM

what is the best method to maintain a browser session without it auto closing?

A browser instance gets killed if it does not get any command for 60 seconds, freeing one instance.

A browser instance gets killed if it does not get any command for 60 seconds, freeing one instance.

sty•1/26/24, 1:35 PM

ahh i got it nvm:

    async alarm() {
        this.keptAliveInSeconds += 10;

        // Extend browser DO life
        if (this.keptAliveInSeconds < KEEP_BROWSER_ALIVE_IN_SECONDS) {

    async alarm() {
        this.keptAliveInSeconds += 10;

        // Extend browser DO life
        if (this.keptAliveInSeconds < KEEP_BROWSER_ALIVE_IN_SECONDS) {

Yacine Hmito•1/27/24, 8:47 AM

I'm trying to hack on @Cloudflare/puppeteer locally. I can't get anything to install this is driving me crazy.

Yacine Hmito•1/27/24, 8:49 AM

I pull the repo, run

npm install

npm install

with node 18 and it fails with "Cannot use import statement outside a module", even though, according to the GitHub Actions workflows, that is supposed to work fine.

Yacine Hmito•1/27/24, 8:49 AM

I tried converting the offending files to

.mts

.mts

but that only moves the problem somwhere else.

Yacine Hmito•1/27/24, 8:51 AM

I tried with the upstream puppeteer on the commit that was forked and the issue is the same verysad

YYacine Hmito I pull the repo, run `npm install` with node 18 and it fails with "Cannot use im...

DaniFoldi•1/27/24, 1:46 PM

Hmm, if the package is esm-only (which I haven't looked into), could it be that you have to be in a module for it to work? Wild guess, all this cjs vs esm nonsense is driving me crazy too, but could you try adding

"type": "module"

"type": "module"

to your package.json?

DDaniFoldi Hmm, if the package is esm-only (which I haven't looked into), could it be that ...

Yacine Hmito•1/29/24, 11:18 AM

The problem is that only the TypeScript files are in ESM. All the rest is in CJS.

Yacine Hmito•1/29/24, 11:35 AM

Sorry to ping you @celso but I didn't manage to run @Cloudflare/puppeteer locally because of the problem I ran into above.
(Context: I am attempting to reproduce the service that manages chromium instances so that I can use the Browser Rendering API locally with workerd)

LLivog Hey! I'm testing out cloudflare/puppeteer api So I m running this code: ``` pag...

jason•1/29/24, 2:24 PM

I'm not sure where that error is coming from, but it may be coming from within your browser context. Buffer isn't defined in the browser's javascript. Maybe the

extractData

extractData

method is trying to use it?

LLivog Thanks for the replay, the extractData is simplifying a object to only get the d...

jason•1/29/24, 2:27 PM

Can you provide the stacktrace of the error?

LLivog Never done that, got any link of instructions on how to do that?

jason•1/29/24, 2:30 PM

Add

console.log(e.stack)

console.log(e.stack)

after the "page on response error"

LLivog Hey! I'm testing out cloudflare/puppeteer api So I m running this code: ``` pag...

Chaika•1/29/24, 2:32 PM

I'm not too familiar with puppeter's APIs, but node support with workers is like this:
node_compat=true in your wrangler.toml polyfills some of the nodejs functionality like buffer, ex

import { Buffer } from 'buffer'

import { Buffer } from 'buffer'

would just work
nodejs_compat flag would only expose it via

node:buffer

node:buffer

, have to specifically use it, ex

import { Buffer } from 'node:buffer'

import { Buffer } from 'node:buffer'

.

What version of cloudflare/puppeter are you using? The latest should work with nodejs_compat flag, but before 0.0.4 release it needed node_compat

Denny•1/29/24, 11:49 PM

Hi - Whats the best way to bypass when websites block us from taking a screenshot?

DDenny Hi - Whats the best way to bypass when websites block us from taking a screensho...

jason•1/30/24, 4:01 AM

1) Request the wishes of the website and don't take the screenshot
2) How exactly are they blocking you?

Jjason 1) Request the wishes of the website and don't take the screenshot 2) How exactl...

Denny•1/30/24, 7:11 AM

Cloudflare antibody firewall

DDenny Cloudflare antibody firewall

jason•1/30/24, 8:52 AM

I'm not sure you are going to get many tips about how to circumvent Cloudflare's Firewall on the Cloudflare Discord...

Ssc Hey @Yacine Any chance you put a minimal example in a gist we could look at?

Yacine Hmito•1/30/24, 12:46 PM

There's no new code. I just pull the repo and did

npm install

npm install

. I checked that I had a compatible version of node set, but it still didn't work.

jason•2/1/24, 6:47 AM

Can someone explain how this works in CF's distributed network? When a browser is started, is it only in a single location? If I wanted browsers in multiple locations, how would I handle that?

Jjason Can someone explain how this works in CF's distributed network? When a browser i...

Pedro•2/1/24, 9:54 AM

Hey jason! Currently it is started on a single location, we'll be adding support for Geolocation, so that for example the content rendered might change according to the location where connection is initiated

jason•2/1/24, 11:18 AM

Thanks as always @Pedro

jason•2/1/24, 2:07 PM

@Pedro Another question... one of the things that excites me most about this project is the hope of scalability. When I've run puppeteer myself on my servers, the challenge that I've always run into it trying to handle scaling it up. Trying to throttle requests to not start too many processes and run the server out of memory, etc. Is one of the goals of this project actually to be able to solve that?

Is this eventually going to work like other Workers do, where I don't need to care about how many requests I throw at it? Or am I going to have to implement all of the throttling code anyways - but now with the limitation that I can't see what's going on inside the server?

Jjason @Pedro Another question... one of the things that excites me most about this pro...

Pedro•2/1/24, 2:10 PM

Great question! We'll be providing a simple way to reuse sessions and keeping them open, so you can focus on the task at hand. We'll still be setting default limits, mostly to prevent abuse and malicious behaviour, but those can be raised according to you use case

PPedro Great question! We'll be providing a simple way to reuse sessions and keeping th...

jason•2/1/24, 2:13 PM

Thanks. Another way to ask the question. Let's say that I have 1000 urls to crawl. Am I going to need to do the work to manually distribute those requests over multiple sessions, throttle them to not send too many at a time, etc? With a normal worker that doesn't use the browser rendering, I would just POST all 1000 requests as fast as possible to my worker and not worry about it. But I suspect with the browser rendering, I wouldn't be able to do that.

Pedro•2/1/24, 2:19 PM

You can choose to reuse the same session (until expired) to crawls the URLs sequentially or you can choose to create concurrent sessions. It's basically up to you. We're providing an easier way to reuse the sessions, but the "puppeteer" behaviour is still up to you to define, so it's flexible enough to accommodate most use cases. (You can already reuse the same session for multiple URLs using Durable Objects as per the tutorial, we're making it easier)

jason•2/1/24, 2:20 PM

Are you saying that a single session can handle one request at a time? So if I wanted to do 1000 requests in parallel, I would need to start 1000 separate sessions?

Jjason Are you saying that a single session can handle one request at a time? So if I w...

Pedro•2/1/24, 2:25 PM

You can use the same session for multiple tasks like crawling multiple URLs. But if you want to do things in parallel, you just create new concurrent sessions. It's like running a Browser and doing the tasks or Multiple Browsers at the same time and performing multiple task in each one at the same time. Hope I was clear enough

jason•2/1/24, 2:31 PM

Yep, I think I understand.

And I think it means that it's not going to have the worker-level scalability that I hoped it would. At least not without costing a fortune. Thanks for the information!

Jjason Yep, I think I understand. 👍 And I think it means that it's not going to have t...

Pedro•2/1/24, 2:53 PM

Would love to learn more about your use case. Please feel free to ping me and let's make sure we have an offer that also suits more specific and demanding use cases such as yours

SSHAH hello, how do i get off the waiting list for this? i've been on it for ages

jason•2/2/24, 5:21 AM

Did you post in #Browser Rendering API Beta Access ? It’s usually within 24 hours.

hmm, semaphore-as-a-service market opportunity

Similar Threads

Similar Threads

Similar Threads