Hey! Is there a way to set userDataDir in puppeteer.launch()?

IIlmerainen Hey! Is there a way to set userDataDir in puppeteer.launch()?

J

jason•3/11/24, 2:03 PM

I would imagine not.

Vvikiival | koda.art I am trying to make this via cloudflare workers. But sadly WebGL does not work i...

D

Deep•3/18/24, 9:08 PM

hey did you ever make some progress on this? i'm doing the same thing for my app.... using 3JS to render some 3D and was even considering setting up my own GPU servers around the US to do my own "edge" rendering.... but would prefer to use cloudflare instead. but, in my research so far the WebGPU support and the browser rendering both don't really have a "real-time 3D rendering" feature as far as i can tell.....

D

Dobroslav•3/20/24, 6:12 PM

Thanks for the access. Is there some roadmap for the full release and increasing rate limits? Current rate limits are not usable, if I want to build product around it.

Or if there is a way to increase these limits, while API is in beta, I would really appreciate it.

Also, is there some pricing available yet?

Thanks!

DDobroslav Thanks for the access. Is there some roadmap for the full release and increasing...

J

jason•3/20/24, 7:29 PM

They've published pricing here: https://blog.cloudflare.com/running-serverless-puppeteer-workers-durable-objects

S

slusatel•3/21/24, 4:19 PM

Whats the best approach for handling SSR with browser rendering, mostly for SEO/social links preview. Would it be Pages middleware that catches and renders pages if bot, and forward the original html if not?
I saw an article somewhere cloudflare already has a similar service, but can't find

W

Wrapper•3/22/24, 9:40 AM

Cloud I get access for browser API please:

cd3066af0931b7b92c5a9dcbb387779d

S

Sam•3/25/24, 2:54 PM

Just wanted to +1 some of the pricing comments - the idea of having a durable object constantly keeping itself alive in order to avoid creating browser sessions seems very strange. I feel I'm in a weird valley of not having consistent-enough traffic to warrant keeping a durable object alive (though I don't know the cost of that on its own), but I'll also probably have >500 requests that need a browser per month at some point. $0.10/session is still very kind pricing, but compared to D1/Workers/KV/R2/etc it seems like a lot (my use case is just PDF generation).

R

rob•3/25/24, 6:33 PM

is there any support for using proxys on the browser rendering api ?

WWrapper Cloud I get access for browser API please: cd3066af0931b7b92c5a9dcbb387779d

J

jason•3/27/24, 8:40 PM

You should post in #Browser Rendering API Beta Access

SSam Just wanted to +1 some of the pricing comments - the idea of having a durable ob...

J

jason•3/27/24, 8:41 PM

There is supposed to be an update coming regarding better session mangement - hopefully some of these concerns will be addressed.

V

villainy•3/28/24, 6:59 PM

Is it possible, if so has anyone tried using the puppeteer fork together with

puppeteer-extra

puppeteer-extra

?

Y

ye•4/1/24, 6:32 AM

has anyone had trouble with the reusing session sample code on https://developers.cloudflare.com/browser-rendering/get-started/reuse-sessions/ ?

Usually the first load works fine as it uses a new session, but if i refresh/etc it doesn't use the old session and instead creates a new session and eventually hits the session limit

I'm creating the second request when the image is returned and the browser is disconnected

Cloudflare Docs

Reuse sessions · Browser Rendering docs

The best way to improve the performance of your browser rendering worker is to reuse sessions. One way to do that is via Durable Objects. Another way …

A

Advany•4/2/24, 10:24 AM

How many resources does a session have? I just tried to do 100 pdf generations with one session (opened 100 pages, loaded content, generated pdf). It did then without any issues?

A

Advany•4/2/24, 10:25 AM

How do I know when I am sending to many resources so I can apply some rate limits

Ssc If you consume too much memory, the browser session will crash. You can look at ...

A

Advany•4/2/24, 5:46 PM

Awesome. Is there a way to force the browser to crash

A

Advany•4/3/24, 11:39 AM

I just want to see if my test setup can 'catch' it

Would It just throw an error by puppeteer?

Z

zegevlier•4/3/24, 2:47 PM

Are there any plans to support WebGL in browser rendering? I was trying to automate taking screenshots of 3d models using (Blockbench)[https://web.blockbench.net/], but it gave an error regarding creating a WebGL context.

Z

zegevlier•4/3/24, 2:47 PM

FWIW you get the same error when trying to use RBI to access blockbench.

G

GenieTheGamer•4/3/24, 7:42 PM

Is there a time limit or an inactive time limit for the beta access? I requested access a month ago and got it, but haven't had a chance to work with the browser api. I finally got a chance to work on it, but I'm still getting an authentication error

G

GenieTheGamer•4/3/24, 7:46 PM

^Nevermind, I was signed into a different account in my terminal

Ssc If you consume too much memory, the browser session will crash. You can look at ...

A

Advany•4/4/24, 5:51 AM

I just discovered the easy way to manage sessions

didnt know about reusing sessions

A

Advany•4/4/24, 9:43 AM

I rewrote the logic for getting session for anyone doing the same. It first tries to connect to a existing session, if that isn't available (anymore), it tries to launch one.

async function getSession(browser) {
    const sessions = await puppeteer.sessions(browser);
    console.log(`getSession: sessions - ${JSON.stringify(sessions)}`);
    const sessionsIds = sessions
        .filter((v) => {
            return !v.connectionId;
        })
        .map((v) => {
            return v.sessionId;
        });

    console.log(`getSession: Available sessionsIds - ${JSON.stringify(sessionsIds)}`);

    if (sessionsIds.length !== 0)
        try {
            const sessionId = sessionsIds[Math.floor(Math.random() * sessionsIds.length)];

            console.log(`getSession: re-using session ${sessionId}`);

            return await puppeteer.connect(browser, sessionId);
        } catch (e) {
            console.log(`getSession: Error connecting to session - Might already be in use. ${JSON.stringify(await puppeteer.sessions(browser))}`);
        }

    try {
        console.log(`getSession: creating new session`);
        return await puppeteer.launch(browser);
    } catch (e) {
        console.log(`getSession: Error creating new session - ${JSON.stringify(await puppeteer.history(browser))}`);
    }
}

async function getSession(browser) {
    const sessions = await puppeteer.sessions(browser);
    console.log(`getSession: sessions - ${JSON.stringify(sessions)}`);
    const sessionsIds = sessions
        .filter((v) => {
            return !v.connectionId;
        })
        .map((v) => {
            return v.sessionId;
        });

    console.log(`getSession: Available sessionsIds - ${JSON.stringify(sessionsIds)}`);

    if (sessionsIds.length !== 0)
        try {
            const sessionId = sessionsIds[Math.floor(Math.random() * sessionsIds.length)];

            console.log(`getSession: re-using session ${sessionId}`);

            return await puppeteer.connect(browser, sessionId);
        } catch (e) {
            console.log(`getSession: Error connecting to session - Might already be in use. ${JSON.stringify(await puppeteer.sessions(browser))}`);
        }

    try {
        console.log(`getSession: creating new session`);
        return await puppeteer.launch(browser);
    } catch (e) {
        console.log(`getSession: Error creating new session - ${JSON.stringify(await puppeteer.history(browser))}`);
    }
}

async function getSession(browser) {
    const sessions = await puppeteer.sessions(browser);
    console.log(`getSession: sessions - ${JSON.stringify(sessions)}`);
    const sessionsIds = sessions
        .filter((v) => {
            return !v.connectionId;
        })
        .map((v) => {
            return v.sessionId;
        });

    console.log(`getSession: Available sessionsIds - ${JSON.stringify(sessionsIds)}`);

    if (sessionsIds.length !== 0)
        try {
            const sessionId = sessionsIds[Math.floor(Math.random() * sessionsIds.length)];

            console.log(`getSession: re-using session ${sessionId}`);

            return await puppeteer.connect(browser, sessionId);
        } catch (e) {
            console.log(`getSession: Error connecting to session - Might already be in use. ${JSON.stringify(await puppeteer.sessions(browser))}`);
        }

    try {
        console.log(`getSession: creating new session`);
        return await puppeteer.launch(browser);
    } catch (e) {
        console.log(`getSession: Error creating new session - ${JSON.stringify(await puppeteer.history(browser))}`);
    }
}

async function getSession(browser) {
    const sessions = await puppeteer.sessions(browser);
    console.log(`getSession: sessions - ${JSON.stringify(sessions)}`);
    const sessionsIds = sessions
        .filter((v) => {
            return !v.connectionId;
        })
        .map((v) => {
            return v.sessionId;
        });

    console.log(`getSession: Available sessionsIds - ${JSON.stringify(sessionsIds)}`);

    if (sessionsIds.length !== 0)
        try {
            const sessionId = sessionsIds[Math.floor(Math.random() * sessionsIds.length)];

            console.log(`getSession: re-using session ${sessionId}`);

            return await puppeteer.connect(browser, sessionId);
        } catch (e) {
            console.log(`getSession: Error connecting to session - Might already be in use. ${JSON.stringify(await puppeteer.sessions(browser))}`);
        }

    try {
        console.log(`getSession: creating new session`);
        return await puppeteer.launch(browser);
    } catch (e) {
        console.log(`getSession: Error creating new session - ${JSON.stringify(await puppeteer.history(browser))}`);
    }
}

O

onnimonni•4/4/24, 11:38 AM

Hey!

I was trying out the Browser Rendering demo from here:
https://developers.cloudflare.com/browser-rendering/platform/puppeteer/

which resulted in following error:

{
    text: 'A request to the Cloudflare API
  (/accounts/XXXXXXXXXXX/workers/scripts/browser-worker/edge-preview) failed.',
    notes: [
      {
        text: 'workers.api.error.no_access_to_browser_worker [code: 10088]'
      }
    ],
    location: undefined,
    kind: 'error',
    code: 10088
  }

{
    text: 'A request to the Cloudflare API
  (/accounts/XXXXXXXXXXX/workers/scripts/browser-worker/edge-preview) failed.',
    notes: [
      {
        text: 'workers.api.error.no_access_to_browser_worker [code: 10088]'
      }
    ],
    location: undefined,
    kind: 'error',
    code: 10088
  }

{
    text: 'A request to the Cloudflare API
  (/accounts/XXXXXXXXXXX/workers/scripts/browser-worker/edge-preview) failed.',
    notes: [
      {
        text: 'workers.api.error.no_access_to_browser_worker [code: 10088]'
      }
    ],
    location: undefined,
    kind: 'error',
    code: 10088
  }

{
    text: 'A request to the Cloudflare API
  (/accounts/XXXXXXXXXXX/workers/scripts/browser-worker/edge-preview) failed.',
    notes: [
      {
        text: 'workers.api.error.no_access_to_browser_worker [code: 10088]'
      }
    ],
    location: undefined,
    kind: 'error',
    code: 10088
  }

And by googling the error I noticed this github issue https://github.com/cloudflare/workers-sdk/issues/3433 which lead me here to apply for the Browser Rendering private beta

O

onnimonni•4/4/24, 11:41 AM

Would there be a chance for me to try this out already? I'm currently starting a new greenfield project using solely CF and there are some cases where the browser automation would be very handy for me.

I've used puppeteer earlier for example when certain payment providers for my company didn't have any API to download monthly accounting reports and we needed to download them by logging into their portal with headless chrome

O

onnimonni•4/4/24, 11:44 AM

Ah sorry for the noice. I didn't notice the separate thread for asking the access: Browser Rendering API Beta Access

A

Advany•4/4/24, 4:27 PM

also in the past we where able to connect to external chromium (as a backup):

const browser = await puppeteer.connect({
    browserWSEndpoint: `wss://chrome.browserless.io/`,
});

const browser = await puppeteer.connect({
    browserWSEndpoint: `wss://chrome.browserless.io/`,
});

thats not working anymore with version 0.0.6

M

Mackenly•4/5/24, 3:05 PM

So with the new GA announcement there are still no changes to limits or pricing right? Just a somewhat cleaner way to handle reusing a session. Doesn't solve the pricing problem at all. PMs were a no show on a research call they requested a few weeks ago and never followed up. Feels like a pattern with this product.

S

Sam•4/5/24, 3:11 PM

Glad there were changes made to session management, but the hard limits are still very concerning - I have more than two environments so it's already a no-go. Would be happy to pay for a "max concurrent browser" increase, it would be a far-cry from the $200/mo for Browserless.

Original message was deleted

M

Mackenly•4/5/24, 3:18 PM

Yes, and that pricing puts you at $4,270 to take a screenshot every minute for 30 days (assuming you don't hackily keep a session open through a durable object alarm).

Original message was deleted

M

Mackenly•4/5/24, 3:19 PM

It's still limited to 60 seconds without interaction. Sessions don't solve the pricing problem.

Original message was deleted

M

Mackenly•4/5/24, 3:23 PM

Yeah in practice every 60s might work within the 60s limit, however let's say it's instead every 61s. (And that's also not considering that it would take a bit for the page to load). Point still stands even if you let's say take a screenshot every 120s, still crazy pricing only avoidable through hacky DO implementations.

A

Advany•4/5/24, 3:47 PM

I am totally ok with pricing. I want higher limits lol. Will have to create multiple accounts for now and proxy the requests around.

T

Thibault•4/5/24, 10:55 PM

I have taken 226 screenshots today

Does it mean I wasted $22.6

TThibault I have taken 226 screenshots today Does it mean I wasted $22.6 😭

M

Mackenly•4/5/24, 11:28 PM

You get 500 included.

MMackenly You get 500 included.

T

Thibault•4/5/24, 11:32 PM

But if I do that daily

R

rob•4/6/24, 1:32 PM

Can you use proxies ?

TThibault I have taken 226 screenshots today Does it mean I wasted $22.6 😭

A

Advany•4/6/24, 7:59 PM

they are working on keeping the session alive for longer. I saw a github branch https://github.com/cloudflare/puppeteer/commits/add_keep_alive_option by @sc

GitHub

Commits · cloudflare/puppeteer

Puppeteer Core fork that works with Cloudflare Browser Workers - Commits · cloudflare/puppeteer

A

Advany•4/6/24, 8:00 PM

but best way is to just use durable objects I think. You can use the same session for parallel requests.

J

James•4/6/24, 10:32 PM

I've filed https://github.com/cloudflare/puppeteer/issues/36 but it would be really good to see some upstream updates for this now that Browser Rendering has hit GA

GitHub

Upstream fixes/features syncing · Issue #36 · cloudflare/puppeteer

It seems like this fork of puppeteer is based on v17.0.0 (released August 2022), whereas the latest version is v22.6.3 (released yesterday, April 2024) Specifically, I'm looking to get deeper J...

C

CalvinTorra•4/7/24, 5:41 PM

do we still need to request access here?

J

James•4/7/24, 5:44 PM

You don't. Browser Rendering is now available on all Workers paid plans

C

CalvinTorra•4/7/24, 6:06 PM

ahhhhh think i missed that in the docs

Hey! Is there a way to set userDataDir in puppeteer.launch()?

Similar Threads