It seems like the @\cloudflare/puppeteer npm package is not working entirely as intended as the regu

It seems like the @\cloudflare/puppeteer npm package is not working entirely as intended as the regular puppeteer does - for example, taking a screenshot of a youtube page, 1st image is using puppeteer on my pc (same result when using a vps I have), and the 2nd image is puppeteer hosted using the browser rendering api on cloudflare using @\cloudflare/puppeteer npm package

For reference, I did use almost exactly the same code, the puppeteer one:

    const browser = await pup.launch({
        headless: `new`,
        defaultViewport: { width: 1920, height: 1080 },
    })

    const getPageSnapshot = (url, fullPage) => new Promise(async (resolve, reject) => { // this returns a promise resolving to a buffer
        const page = await browser.newPage().catch(reject)
        await page.goto(url).catch(reject)

        resolve(await page.screenshot({
            fullPage
        }).then(async (r) => { await page.close(); return r }).catch(reject))
    })

    const browser = await pup.launch({
        headless: `new`,
        defaultViewport: { width: 1920, height: 1080 },
    })

    const getPageSnapshot = (url, fullPage) => new Promise(async (resolve, reject) => { // this returns a promise resolving to a buffer
        const page = await browser.newPage().catch(reject)
        await page.goto(url).catch(reject)

        resolve(await page.screenshot({
            fullPage
        }).then(async (r) => { await page.close(); return r }).catch(reject))
    })

    const browser = await pup.launch({
        headless: `new`,
        defaultViewport: { width: 1920, height: 1080 },
    })

    const getPageSnapshot = (url, fullPage) => new Promise(async (resolve, reject) => { // this returns a promise resolving to a buffer
        const page = await browser.newPage().catch(reject)
        await page.goto(url).catch(reject)

        resolve(await page.screenshot({
            fullPage
        }).then(async (r) => { await page.close(); return r }).catch(reject))
    })

    const browser = await pup.launch({
        headless: `new`,
        defaultViewport: { width: 1920, height: 1080 },
    })

    const getPageSnapshot = (url, fullPage) => new Promise(async (resolve, reject) => { // this returns a promise resolving to a buffer
        const page = await browser.newPage().catch(reject)
        await page.goto(url).catch(reject)

        resolve(await page.screenshot({
            fullPage
        }).then(async (r) => { await page.close(); return r }).catch(reject))
    })

And the cloudflare one:

    async handle(
        request: Request,
        env: any,
        context: any,
        data: Record<string, any>
    ) : Promise<Response> {
                // some url handling
        
        const browser = await puppeteer.launch(env.MYBROWSER);
        const page = await browser.newPage();
        await page.setViewport({ width: 1920, height: 1080 })
        await page.goto(url);
        const screenshot = (await page.screenshot({
            fullPage: false,
        })) as Buffer;
        await browser.close();

        return new Response(screenshot, {
            headers: {
                'Content-Type': `image/jpeg`,
            },
        });
    }

    async handle(
        request: Request,
        env: any,
        context: any,
        data: Record<string, any>
    ) : Promise<Response> {
                // some url handling
        
        const browser = await puppeteer.launch(env.MYBROWSER);
        const page = await browser.newPage();
        await page.setViewport({ width: 1920, height: 1080 })
        await page.goto(url);
        const screenshot = (await page.screenshot({
            fullPage: false,
        })) as Buffer;
        await browser.close();

        return new Response(screenshot, {
            headers: {
                'Content-Type': `image/jpeg`,
            },
        });
    }

    async handle(
        request: Request,
        env: any,
        context: any,
        data: Record<string, any>
    ) : Promise<Response> {
                // some url handling
        
        const browser = await puppeteer.launch(env.MYBROWSER);
        const page = await browser.newPage();
        await page.setViewport({ width: 1920, height: 1080 })
        await page.goto(url);
        const screenshot = (await page.screenshot({
            fullPage: false,
        })) as Buffer;
        await browser.close();

        return new Response(screenshot, {
            headers: {
                'Content-Type': `image/jpeg`,
            },
        });
    }

    async handle(
        request: Request,
        env: any,
        context: any,
        data: Record<string, any>
    ) : Promise<Response> {
                // some url handling
        
        const browser = await puppeteer.launch(env.MYBROWSER);
        const page = await browser.newPage();
        await page.setViewport({ width: 1920, height: 1080 })
        await page.goto(url);
        const screenshot = (await page.screenshot({
            fullPage: false,
        })) as Buffer;
        await browser.close();

        return new Response(screenshot, {
            headers: {
                'Content-Type': `image/jpeg`,
            },
        });
    }

PPaige It seems like the @\cloudflare/puppeteer npm package is not working entirely as ...

Unsmart•12/4/23, 3:31 PM

try adding a waitUntil "load" on the page goto

UUnsmart try adding a waitUntil "load" on the page goto

PaigeOP•12/4/23, 3:39 PM

using

        await page.goto(url, {
            waitUntil: `load`
        });

        await page.goto(url, {
            waitUntil: `load`
        });

gives an identical result

Jürgen B•12/5/23, 9:52 AM

I'm getting the following error "Exceeded Resources" in my logs for my scheduled execs, however it seems my script works just great, any ideas?

PaigeOP•12/5/23, 7:59 PM

will try

PaigeOP•12/6/23, 6:39 AM

In the docs there was a screenshot hinting at some additional pricing per opened browser session, but I cannot find any actual pricing page for this, does it cost me anything extra to open browser instances, and where can I find how much?

dj_urgarden•12/6/23, 1:19 PM

Hey hello there, can I please get an access for

291b861f09df80526646856f248a8776

291b861f09df80526646856f248a8776

PPaige In the docs there was a screenshot hinting at some additional pricing per opened...

Mackenly•12/6/23, 2:59 PM

The pricing is a big problem with this service. It was announced here: https://blog.cloudflare.com/running-serverless-puppeteer-workers-durable-objects/

The Cloudflare Blog

Running Serverless Puppeteer with Workers and Durable Objects

We’ve heard from developers that configuring and maintaining their own serverless browser automation systems can be quite painful. The Workers Browser Rendering API solves this

MMackenly The pricing is a big problem with this service. It was announced here: https://b...

PaigeOP•12/6/23, 3:48 PM

So in theory I could just keep 10 concurrent sessions as durable objects and use those together for all of my requests?

PaigeOP•12/6/23, 3:48 PM

for just the 5$/month

PPaige So in theory I could just keep 10 concurrent sessions as durable objects and use...

Mackenly•12/6/23, 3:52 PM

Yeah in theory. There are reasons they'll get shut down (updates and stuff) but in theory, yes.

MMackenly Yeah in theory. There are reasons they'll get shut down (updates and stuff) but ...

PaigeOP•12/6/23, 3:57 PM

do you coincidentally know an estimate of how often that'd typically be?

PPaige do you coincidentally know an estimate of how often that'd typically be?

Mackenly•12/6/23, 10:45 PM

I'm too hesitant to use it in prod because of the pricing risk (it's beta still anyways) so don't have any real prod use to compare the price.

PaigeOP•12/7/23, 6:39 AM

Yeah I came to the same conclusion

Jiamin•12/9/23, 3:14 AM

Hello Cloudflare teams

Jiamin•12/9/23, 3:15 AM

Can I have the ability to access the Cloudflare Browser Rendering API? I want to use OpenAI GPTs to call a worker and achieve a cool functionality.

Jiamin•12/9/23, 3:18 AM

Can I get access? emma

afe2cf41c6b92b459a3b7e502156ca9b

Erisa•12/9/23, 3:26 AM

You'll need this thread browser-rendering-apiBrowser Rendering API Beta Access

EErisa You'll need this thread https://discord.com/channels/595317990191398933/11126750...

Jiamin•12/9/23, 3:30 AM

Dragon•12/9/23, 8:25 AM

Hi Cloudflare folks, I am the founder of MultiOn AI (https://www.multion.ai/) - building general-purpose web AI agents, we are looking to scale our browser workloads and will love to try out the serverless pupeteer with Cloudflare for our production loads

.

Let me know if some one from the team will be available to get us set up and get us early access! I am at div@multion.ai

MultiOn - Your Personal AI Agent

Your personal AI agent

MMackenly I'm too hesitant to use it in prod because of the pricing risk (it's beta still ...

Pedro•12/11/23, 1:59 PM

Hey @Mackenly Jones, we mentioned here previously, but we're working on addressing the pricing concerns by making the session management easier

DDragon Hi Cloudflare folks, I am the founder of MultiOn AI (https://www.multion.ai/) -...

Pedro•12/11/23, 2:00 PM

Hey @dragon, Looking forward to learning more. Please let us know if there anything we can help out with

Vikash•12/12/23, 8:26 AM

I was testing to capture screenshot for top 5k top domains from CF - https://radar.cloudflare.com/domains to test if we can migrate our Screenshot API from AWS (https://agenty.com/tools/webpage-to-screenshot)

I created a queue to send request to D0 to capture screenshot and upload

But the queue is getting error randomly :

Unabled to create new browser:

Unabled to create new browser:

  {
      "message": [
        "Browser DO: Could not start browser instance. Error: Error: Unabled to create new browser: code: 429: message: Too Many Requests",
        "Error: Unabled to create new browser: code: 429: message: Too Many Requests\n    at PuppeteerWorkers.launch (index.js:19201:13)\n    at async Browser2.fetch (index.js:43967:24)"
      ],
      "level": "error",
      "timestamp": 1702366528655
    }

  {
      "message": [
        "Browser DO: Could not start browser instance. Error: Error: Unabled to create new browser: code: 429: message: Too Many Requests",
        "Error: Unabled to create new browser: code: 429: message: Too Many Requests\n    at PuppeteerWorkers.launch (index.js:19201:13)\n    at async Browser2.fetch (index.js:43967:24)"
      ],
      "level": "error",
      "timestamp": 1702366528655
    }

So it sems the D1 browser reset automatically and the new launch fails to create the browser.

DO Code

  async fetch(url: string) {
    //if there's a browser session open, re-use it
    if (!this.browser || !this.browser.isConnected()) {
      console.log(`Browser DO: Starting new instance`);
      try {
        this.browser = await puppeteer.launch(this.env.CHROMIUM);
      } catch (e) {
        console.error(
          `Browser DO: Could not start browser instance. Error: ${e}`,
          e.stack
        );
        throw e;
      }
    }
   .......
  }

  async fetch(url: string) {
    //if there's a browser session open, re-use it
    if (!this.browser || !this.browser.isConnected()) {
      console.log(`Browser DO: Starting new instance`);
      try {
        this.browser = await puppeteer.launch(this.env.CHROMIUM);
      } catch (e) {
        console.error(
          `Browser DO: Could not start browser instance. Error: ${e}`,
          e.stack
        );
        throw e;
      }
    }
   .......
  }

Cloudflare Radar

Up to date Internet trends and insight

Agenty

Agenty - No-Code Web Automation Software

Automate your business with advanced, fully-featured agents on Agenty. Fast, scalable and no-code web automation tool for web scraping, change monitoring, browser automation and more...

Ssc Looks like your browser is being closed under you and so you're hitting rate lim...

Vikash•12/12/23, 9:52 AM

You mean here?

 async alarm() {
    this.keptAliveInSeconds += 10;

    // Extend browser DO life
    if (this.keptAliveInSeconds < KEEP_BROWSER_ALIVE_IN_SECONDS) {
      console.log(
        `Browser DO: has been kept alive for ${this.keptAliveInSeconds} seconds. Extending lifespan.`
      );
      await this.storage.setAlarm(Date.now() + 10 * 1000);
      await this.browser.version();
    } else
      console.log(
        `Browser DO: exceeded life of ${KEEP_BROWSER_ALIVE_IN_SECONDS}. Browser DO will be shut down in 10 seconds.`
      );
  }

 async alarm() {
    this.keptAliveInSeconds += 10;

    // Extend browser DO life
    if (this.keptAliveInSeconds < KEEP_BROWSER_ALIVE_IN_SECONDS) {
      console.log(
        `Browser DO: has been kept alive for ${this.keptAliveInSeconds} seconds. Extending lifespan.`
      );
      await this.storage.setAlarm(Date.now() + 10 * 1000);
      await this.browser.version();
    } else
      console.log(
        `Browser DO: exceeded life of ${KEEP_BROWSER_ALIVE_IN_SECONDS}. Browser DO will be shut down in 10 seconds.`
      );
  }

Original message was deleted

zegevlier•12/14/23, 4:02 PM

Browser Rendering API Beta Access

vikiival | koda.art•12/21/23, 12:24 PM

Hello @zegevlier @sc can you please help me with debugging?

vikiival | koda.art•12/21/23, 12:24 PM

Seems my original message was deleted

zegevlier•12/21/23, 12:24 PM

?pings

Flare•12/21/23, 12:24 PM

Please do not ping community members for non-moderation reasons. Doing so will not solve your issue faster and will make people less likely to want to help you.

Zzegevlier ?pings

vikiival | koda.art•12/21/23, 12:25 PM

Just want to know why my message was deleted

zegevlier•12/21/23, 12:26 PM

I wasn't the one who deleted it, but I see that you advertised giving money for someone to help you. That's not allowed here.

Zzegevlier I wasn't the one who deleted it, but I see that you advertised giving money for ...

vikiival | koda.art•12/21/23, 1:02 PM

"motivating someone to help me ;D"

But noted.
Still are there some limitations on the Browser rendering ?

Vvikiival | koda.art "motivating someone to help me ;D" But noted. Still are there some limitations...

Pedro•12/22/23, 3:39 PM

Hi! You can check the current limits here: https://developers.cloudflare.com/browser-rendering/platform/limits/ . Would love to learn more about your use-case and your feedback on the current offer.

Limits · Browser Rendering docs

The Workers Browser Rendering API is currently in beta. Add yourself to the waitlist to reserve your spot, and as we expand availability, we will …

kian•12/30/23, 10:34 PM

Seems like Node/Undici types instead of workers-types

James•12/30/23, 10:42 PM

You might be hitting this: https://github.com/cloudflare/workerd/issues/1298

xeon06•12/30/23, 11:07 PM

Oops, didn't realize this was in beta, I wrote all the code already and realize I can't run it

d85a0daa3e2452f1bcc7ba56d59ceb5e

d85a0daa3e2452f1bcc7ba56d59ceb5e

in case anyone is feeling generous with the beta access

kian•12/30/23, 11:09 PM

Post it in Browser Rendering API Beta Access and it'll likely be seen quicker

xeon06•12/30/23, 11:10 PM

Missed that thank you!

PPedro Hi! You can check the current limits here: https://developers.cloudflare.com/bro...

vikiival | koda.art•1/2/24, 10:33 AM

I am trying to make this via cloudflare workers. But sadly WebGL does not work in cloudflare

https://dev.to/rainerselvet/serverless-3d-webgl-rendering-with-threejs-4o11

DEV Community

Serverless 3D WebGL rendering with ThreeJS

This image above was rendered in a serverless function on page load (not kidding, check the image...

Vvikiival | koda.art I am trying to make this via cloudflare workers. But sadly WebGL does not work i...

Pedro•1/2/24, 11:42 AM

Hey @vikiival | KodaDot , I believe you're right, but since that's not directly related to Browser Rendering, I would encourage you to post your question on the Workers channel in order to get a better answer regarding what you're trying to accomplish with WebGL

jason•1/8/24, 7:36 AM

@Pedro You had mentioned that the team is working on updates to Session handling. Can you give us an idea on when those updates might be available? I understand that you probably can't be very specific, but are we thinking weeks? Months? Q1? Q2?. Can you give any sort of estimate?

Pedro•1/8/24, 8:30 AM

Hey

Jjason @Pedro You had mentioned that the team is working on updates to Session handling...

Pedro•1/8/24, 8:33 AM

Hey! if nothing unexpected comes up, late Q1

If you have feedback or ideas on how this should ideally work, please do pass them along

PPedro Hey! if nothing unexpected comes up, late Q1 🙏 If you have feedback or ideas o...

jason•1/8/24, 8:48 AM

What I would like is to be able to start a "Durable Browser" and be charged per "Durable Browser" session started. CF would then handle all that behind the scenes stuff to keep the browser open and running (and restarting after crashes, or memory leaks, or whatever) and I don't have to worry about it. And then I can know in my code that I'm only registering 1 or 5 or 10, and then I can manage it accordingly. I'm not sure how that would work from a scalability perspective. But I can't stomach the volatility of either being charged pennies or thousands of dollars because of a memory leak in the browser, which I can't control, crashing and restarting. Or even things like deploying new code or attaching to the log viewer; if I understand right, my durable objects get reset; which starts a new browser session.

jason•1/8/24, 8:49 AM

I guess that's not super helpful. All in all, I don't really care about implementation details. I just need predictability in the pricing.

Jjason What I would like is to be able to start a "Durable Browser" and be charged per ...

Pedro•1/8/24, 9:05 AM

That's really helpful and aligned we our concerns. Thanks

captain•1/8/24, 7:45 PM

questions about sessions and number of pages and geography. I saw that for $5 a month you get 500 sessions. I was wanting to check multiple search results on certain google searches. 1. Is there a way to define the cloudflare geographic node making the remote request. 2. Is there a limit to how many pages I could view in a single session? 3. Can I block this service from other cloudflare customers using this on my domains? 4. (general puppeteer question..can't find a conclusive answer in google) Is there a good suggestion on how to add a simple timestamp to screen shot with the lowest overhead?

Ccaptain questions about sessions and number of pages and geography. I saw that for $5 a ...

captain•1/8/24, 11:32 PM

(for #4, I injected a new div node with the time stamp before taking a screenshot and that solved my issue!) Now reading back about needing to create and keep open a durable object to maintain a single session if necessary to avoid going over 500 and seeing that it says open as long as there is not a empty 60 seconds. SO if I had had a cront job that checked perhaps a 100 pages with a sleep delay of 3 seconds to allow the content to load before doing a screen shot, and say this cron runs once a day then theoretically it can run 15 minutes giving me the opportuity to run 15 * 60 / 3 = 300 pages in a single cron job max per session

It seems like the @\cloudflare/puppeteer npm package is not working entirely as intended as the regu

Similar Threads

Similar Threads

Similar Threads