Hi Cloudflare folks, I am the founder of MultiOn AI (https://www.multion.ai/) - building general-pu

MMackenly I'm too hesitant to use it in prod because of the pricing risk (it's beta still ...

P

Pedro•12/11/23, 1:59 PM

Hey @Mackenly Jones, we mentioned here previously, but we're working on addressing the pricing concerns by making the session management easier

DDragon Hi Cloudflare folks, I am the founder of MultiOn AI (https://www.multion.ai/) -...

P

Pedro•12/11/23, 2:00 PM

Hey @dragon, Looking forward to learning more. Please let us know if there anything we can help out with

V

Vikash•12/12/23, 8:26 AM

I was testing to capture screenshot for top 5k top domains from CF - https://radar.cloudflare.com/domains to test if we can migrate our Screenshot API from AWS (https://agenty.com/tools/webpage-to-screenshot)

I created a queue to send request to D0 to capture screenshot and upload
But the queue is getting error randomly :
```
Unabled to create new browser:
```
Unabled to create new browser:
```
Unabled to create new browser:
```
Unabled to create new browser:

  {
      "message": [
        "Browser DO: Could not start browser instance. Error: Error: Unabled to create new browser: code: 429: message: Too Many Requests",
        "Error: Unabled to create new browser: code: 429: message: Too Many Requests\n    at PuppeteerWorkers.launch (index.js:19201:13)\n    at async Browser2.fetch (index.js:43967:24)"
      ],
      "level": "error",
      "timestamp": 1702366528655
    }

  {
      "message": [
        "Browser DO: Could not start browser instance. Error: Error: Unabled to create new browser: code: 429: message: Too Many Requests",
        "Error: Unabled to create new browser: code: 429: message: Too Many Requests\n    at PuppeteerWorkers.launch (index.js:19201:13)\n    at async Browser2.fetch (index.js:43967:24)"
      ],
      "level": "error",
      "timestamp": 1702366528655
    }

  {
      "message": [
        "Browser DO: Could not start browser instance. Error: Error: Unabled to create new browser: code: 429: message: Too Many Requests",
        "Error: Unabled to create new browser: code: 429: message: Too Many Requests\n    at PuppeteerWorkers.launch (index.js:19201:13)\n    at async Browser2.fetch (index.js:43967:24)"
      ],
      "level": "error",
      "timestamp": 1702366528655
    }

  {
      "message": [
        "Browser DO: Could not start browser instance. Error: Error: Unabled to create new browser: code: 429: message: Too Many Requests",
        "Error: Unabled to create new browser: code: 429: message: Too Many Requests\n    at PuppeteerWorkers.launch (index.js:19201:13)\n    at async Browser2.fetch (index.js:43967:24)"
      ],
      "level": "error",
      "timestamp": 1702366528655
    }

So it sems the D1 browser reset automatically and the new launch fails to create the browser.

DO Code

  async fetch(url: string) {
    //if there's a browser session open, re-use it
    if (!this.browser || !this.browser.isConnected()) {
      console.log(`Browser DO: Starting new instance`);
      try {
        this.browser = await puppeteer.launch(this.env.CHROMIUM);
      } catch (e) {
        console.error(
          `Browser DO: Could not start browser instance. Error: ${e}`,
          e.stack
        );
        throw e;
      }
    }
   .......
  }

  async fetch(url: string) {
    //if there's a browser session open, re-use it
    if (!this.browser || !this.browser.isConnected()) {
      console.log(`Browser DO: Starting new instance`);
      try {
        this.browser = await puppeteer.launch(this.env.CHROMIUM);
      } catch (e) {
        console.error(
          `Browser DO: Could not start browser instance. Error: ${e}`,
          e.stack
        );
        throw e;
      }
    }
   .......
  }

  async fetch(url: string) {
    //if there's a browser session open, re-use it
    if (!this.browser || !this.browser.isConnected()) {
      console.log(`Browser DO: Starting new instance`);
      try {
        this.browser = await puppeteer.launch(this.env.CHROMIUM);
      } catch (e) {
        console.error(
          `Browser DO: Could not start browser instance. Error: ${e}`,
          e.stack
        );
        throw e;
      }
    }
   .......
  }

  async fetch(url: string) {
    //if there's a browser session open, re-use it
    if (!this.browser || !this.browser.isConnected()) {
      console.log(`Browser DO: Starting new instance`);
      try {
        this.browser = await puppeteer.launch(this.env.CHROMIUM);
      } catch (e) {
        console.error(
          `Browser DO: Could not start browser instance. Error: ${e}`,
          e.stack
        );
        throw e;
      }
    }
   .......
  }

Cloudflare Radar

Up to date Internet trends and insight

Agenty

Agenty - No-Code Web Automation Software

Automate your business with advanced, fully-featured agents on Agenty. Fast, scalable and no-code web automation tool for web scraping, change monitoring, browser automation and more...

Ssc Looks like your browser is being closed under you and so you're hitting rate lim...

V

Vikash•12/12/23, 9:52 AM

You mean here?

 async alarm() {
    this.keptAliveInSeconds += 10;

    // Extend browser DO life
    if (this.keptAliveInSeconds < KEEP_BROWSER_ALIVE_IN_SECONDS) {
      console.log(
        `Browser DO: has been kept alive for ${this.keptAliveInSeconds} seconds. Extending lifespan.`
      );
      await this.storage.setAlarm(Date.now() + 10 * 1000);
      await this.browser.version();
    } else
      console.log(
        `Browser DO: exceeded life of ${KEEP_BROWSER_ALIVE_IN_SECONDS}. Browser DO will be shut down in 10 seconds.`
      );
  }

 async alarm() {
    this.keptAliveInSeconds += 10;

    // Extend browser DO life
    if (this.keptAliveInSeconds < KEEP_BROWSER_ALIVE_IN_SECONDS) {
      console.log(
        `Browser DO: has been kept alive for ${this.keptAliveInSeconds} seconds. Extending lifespan.`
      );
      await this.storage.setAlarm(Date.now() + 10 * 1000);
      await this.browser.version();
    } else
      console.log(
        `Browser DO: exceeded life of ${KEEP_BROWSER_ALIVE_IN_SECONDS}. Browser DO will be shut down in 10 seconds.`
      );
  }

 async alarm() {
    this.keptAliveInSeconds += 10;

    // Extend browser DO life
    if (this.keptAliveInSeconds < KEEP_BROWSER_ALIVE_IN_SECONDS) {
      console.log(
        `Browser DO: has been kept alive for ${this.keptAliveInSeconds} seconds. Extending lifespan.`
      );
      await this.storage.setAlarm(Date.now() + 10 * 1000);
      await this.browser.version();
    } else
      console.log(
        `Browser DO: exceeded life of ${KEEP_BROWSER_ALIVE_IN_SECONDS}. Browser DO will be shut down in 10 seconds.`
      );
  }

 async alarm() {
    this.keptAliveInSeconds += 10;

    // Extend browser DO life
    if (this.keptAliveInSeconds < KEEP_BROWSER_ALIVE_IN_SECONDS) {
      console.log(
        `Browser DO: has been kept alive for ${this.keptAliveInSeconds} seconds. Extending lifespan.`
      );
      await this.storage.setAlarm(Date.now() + 10 * 1000);
      await this.browser.version();
    } else
      console.log(
        `Browser DO: exceeded life of ${KEEP_BROWSER_ALIVE_IN_SECONDS}. Browser DO will be shut down in 10 seconds.`
      );
  }

Original message was deleted

Z

zegevlier•12/14/23, 4:02 PM

Browser Rendering API Beta Access

V

vikiival | koda.art•12/21/23, 12:24 PM

Hello @zegevlier @sc can you please help me with debugging?

V

vikiival | koda.art•12/21/23, 12:24 PM

Seems my original message was deleted

Z

zegevlier•12/21/23, 12:24 PM

?pings

F

Flare•12/21/23, 12:24 PM

Please do not ping community members for non-moderation reasons. Doing so will not solve your issue faster and will make people less likely to want to help you.

Zzegevlier ?pings

V

vikiival | koda.art•12/21/23, 12:25 PM

Just want to know why my message was deleted

Z

zegevlier•12/21/23, 12:26 PM

I wasn't the one who deleted it, but I see that you advertised giving money for someone to help you. That's not allowed here.

Zzegevlier I wasn't the one who deleted it, but I see that you advertised giving money for ...

V

vikiival | koda.art•12/21/23, 1:02 PM

"motivating someone to help me ;D"

But noted.
Still are there some limitations on the Browser rendering ?

Vvikiival | koda.art "motivating someone to help me ;D" But noted. Still are there some limitations...

P

Pedro•12/22/23, 3:39 PM

Hi! You can check the current limits here: https://developers.cloudflare.com/browser-rendering/platform/limits/ . Would love to learn more about your use-case and your feedback on the current offer.

Limits · Browser Rendering docs

The Workers Browser Rendering API is currently in beta. Add yourself to the waitlist to reserve your spot, and as we expand availability, we will …

K

kian•12/30/23, 10:34 PM

Seems like Node/Undici types instead of workers-types

J

James•12/30/23, 10:42 PM

You might be hitting this: https://github.com/cloudflare/workerd/issues/1298

X

xeon06•12/30/23, 11:07 PM

Oops, didn't realize this was in beta, I wrote all the code already and realize I can't run it

d85a0daa3e2452f1bcc7ba56d59ceb5e

d85a0daa3e2452f1bcc7ba56d59ceb5e

d85a0daa3e2452f1bcc7ba56d59ceb5e

d85a0daa3e2452f1bcc7ba56d59ceb5e in case anyone is feeling generous with the beta access

K

kian•12/30/23, 11:09 PM

Post it in Browser Rendering API Beta Access and it'll likely be seen quicker

X

xeon06•12/30/23, 11:10 PM

Missed that thank you!

PPedro Hi! You can check the current limits here: https://developers.cloudflare.com/bro...

V

vikiival | koda.art•1/2/24, 10:33 AM

I am trying to make this via cloudflare workers. But sadly WebGL does not work in cloudflare

https://dev.to/rainerselvet/serverless-3d-webgl-rendering-with-threejs-4o11

DEV Community

Serverless 3D WebGL rendering with ThreeJS

This image above was rendered in a serverless function on page load (not kidding, check the image...

Vvikiival | koda.art I am trying to make this via cloudflare workers. But sadly WebGL does not work i...

P

Pedro•1/2/24, 11:42 AM

Hey @vikiival | KodaDot , I believe you're right, but since that's not directly related to Browser Rendering, I would encourage you to post your question on the Workers channel in order to get a better answer regarding what you're trying to accomplish with WebGL

J

jason•1/8/24, 7:36 AM

@Pedro You had mentioned that the team is working on updates to Session handling. Can you give us an idea on when those updates might be available? I understand that you probably can't be very specific, but are we thinking weeks? Months? Q1? Q2?. Can you give any sort of estimate?

P

Pedro•1/8/24, 8:30 AM

Hey

Jjason @Pedro You had mentioned that the team is working on updates to Session handling...

P

Pedro•1/8/24, 8:33 AM

Hey! if nothing unexpected comes up, late Q1

If you have feedback or ideas on how this should ideally work, please do pass them along

PPedro Hey! if nothing unexpected comes up, late Q1 🙏 If you have feedback or ideas o...

J

jason•1/8/24, 8:48 AM

What I would like is to be able to start a "Durable Browser" and be charged per "Durable Browser" session started. CF would then handle all that behind the scenes stuff to keep the browser open and running (and restarting after crashes, or memory leaks, or whatever) and I don't have to worry about it. And then I can know in my code that I'm only registering 1 or 5 or 10, and then I can manage it accordingly. I'm not sure how that would work from a scalability perspective. But I can't stomach the volatility of either being charged pennies or thousands of dollars because of a memory leak in the browser, which I can't control, crashing and restarting. Or even things like deploying new code or attaching to the log viewer; if I understand right, my durable objects get reset; which starts a new browser session.

J

jason•1/8/24, 8:49 AM

I guess that's not super helpful. All in all, I don't really care about implementation details. I just need predictability in the pricing.

Jjason What I would like is to be able to start a "Durable Browser" and be charged per ...

P

Pedro•1/8/24, 9:05 AM

That's really helpful and aligned we our concerns. Thanks

C

captain•1/8/24, 7:45 PM

questions about sessions and number of pages and geography. I saw that for $5 a month you get 500 sessions. I was wanting to check multiple search results on certain google searches. 1. Is there a way to define the cloudflare geographic node making the remote request. 2. Is there a limit to how many pages I could view in a single session? 3. Can I block this service from other cloudflare customers using this on my domains? 4. (general puppeteer question..can't find a conclusive answer in google) Is there a good suggestion on how to add a simple timestamp to screen shot with the lowest overhead?

Ccaptain questions about sessions and number of pages and geography. I saw that for $5 a ...

C

captain•1/8/24, 11:32 PM

(for #4, I injected a new div node with the time stamp before taking a screenshot and that solved my issue!) Now reading back about needing to create and keep open a durable object to maintain a single session if necessary to avoid going over 500 and seeing that it says open as long as there is not a empty 60 seconds. SO if I had had a cront job that checked perhaps a 100 pages with a sleep delay of 3 seconds to allow the content to load before doing a screen shot, and say this cron runs once a day then theoretically it can run 15 minutes giving me the opportuity to run 15 * 60 / 3 = 300 pages in a single cron job max per session

P

Pedro•1/9/24, 8:07 AM

Hey @captain, you can indeed keep a session open using durable objects and check multiple search results with the same session. Please find an example here: https://developers.cloudflare.com/browser-rendering/get-started/browser-rendering-with-do/

We are making changed to the product so that in the future you won't have to manage the DO and keepint it alive. We also have plans to add geolocation, so you can choose where the request will be originated, but it is not yet available in current beta. Thanks for sharing your solution for the timestamp

Deploy a Browser Rendering Worker with Durable Objects · Browser Re...

By following this guide, you will create a Worker that uses the Browser Rendering API along with Durable Objects to take screenshots from web pages …

PPedro Hey @captain, you can indeed keep a session open using durable objects and check...

C

captain•1/9/24, 4:32 PM

thanks pedro for the update that's great to hear geo location and not have to manage a DO. I dont want to maintain a DO as its one more thing to think about (and haven't played with a DO yet...so far I'm rocking the D1 database and R2 storage)

E

ExSidius•1/10/24, 4:42 PM

Hey guys - I've signed up for the waitlist. Is there anything else I need to do to get access to the beta?

Z

zegevlier•1/10/24, 4:54 PM

There is! Put your account ID in this thread: browser-rendering-apiBrowser Rendering API Beta Access

C

captain•1/13/24, 5:58 AM

why is puppetteer seeming way faster within the last 15 min?

Jjason Hello. I'm taking a screenshot of websites with the `screenshot()` method, and e...

G

georgehenderson•1/13/24, 4:55 PM

@jason did you figure out how to make it not take a screenshot of the full page?

G

georgehenderson•1/13/24, 5:12 PM

I was able to work around this bug by manually clipping the viewport which is a bit annoying but works.

page.screenshot({ clip: { x: 0, y: lastScroll || 0, width: data.innerWidth, height: data.innerHeight } })

page.screenshot({ clip: { x: 0, y: lastScroll || 0, width: data.innerWidth, height: data.innerHeight } })

page.screenshot({ clip: { x: 0, y: lastScroll || 0, width: data.innerWidth, height: data.innerHeight } })

page.screenshot({ clip: { x: 0, y: lastScroll || 0, width: data.innerWidth, height: data.innerHeight } })

G

georgehenderson•1/16/24, 1:26 PM

this does work, but scrollbars appear

U

usmannk•1/16/24, 2:04 PM

so.. concurrency is very bad when using browser rendering DOs right? If two requests occur simultaneously, one will often overwrite the puppeteer "page" while the other is waiting for page load. What's the recommended way to avoid this? Durable Object input gates are not suitable for this purpose

U

usmannk•1/16/24, 2:05 PM

is

blockConcurrencyWhile

blockConcurrencyWhile

blockConcurrencyWhile

blockConcurrencyWhile the way to go?

Z

zegevlier•1/16/24, 2:06 PM

I was looking up the docs link for that haha. Yea, I believe that would work here. Alternatively, there's #queues-beta.

U

usmannk•1/16/24, 2:07 PM

yeah that makes sense, I was also looking at queues and noticed they have customizable concurrency

Z

zegevlier•1/16/24, 2:08 PM

Although I'm not entirely sure it's needed? Haven't played around with it too much, but would you not be able to get the existing browser, then create a new page, and use that page? I don't think that overwrites it?

U

usmannk•1/16/24, 2:08 PM

you're right, but eventually you'd have too many tabs open

U

usmannk•1/16/24, 2:08 PM

tabs == pages

Z

zegevlier•1/16/24, 2:09 PM

Yea that's true, although if you use something like queues you could prevent it from going over the specified concurrency

U

usmannk•1/16/24, 2:09 PM

yeah, but im not sure how to figure out which page is safe to use

U

usmannk•1/16/24, 2:09 PM

even if concurrency is limited to, say, 5

Z

zegevlier•1/16/24, 2:10 PM

Why not have each process create a new page, then close it when it's done?

Hi Cloudflare folks, I am the founder of MultiOn AI (https://www.multion.ai/) - building general-pu

Similar Threads