I pull the repo, run `npm install` with node 18 and it fails with "Cannot use import statement outsi

I pull the repo, run

npm install

npm install

with node 18 and it fails with "Cannot use import statement outside a module", even though, according to the GitHub Actions workflows, that is supposed to work fine.

Yacine HmitoOP•1/27/24, 8:49 AM

I tried converting the offending files to .mts.mts but that only moves the problem somwhere else.

Yacine HmitoOP•1/27/24, 8:51 AM

I tried with the upstream puppeteer on the commit that was forked and the issue is the same verysad

YYacine Hmito I pull the repo, run `npm install` with node 18 and it fails with "Cannot use im...

DaniFoldi•1/27/24, 1:46 PM

Hmm, if the package is esm-only (which I haven't looked into), could it be that you have to be in a module for it to work? Wild guess, all this cjs vs esm nonsense is driving me crazy too, but could you try adding "type": "module""type": "module" to your package.json?

DDaniFoldi Hmm, if the package is esm-only (which I haven't looked into), could it be that ...

Yacine HmitoOP•1/29/24, 11:18 AM

The problem is that only the TypeScript files are in ESM. All the rest is in CJS.

Yacine HmitoOP•1/29/24, 11:35 AM

Sorry to ping you @celso but I didn't manage to run @Cloudflare/puppeteer locally because of the problem I ran into above.
(Context: I am attempting to reproduce the service that manages chromium instances so that I can use the Browser Rendering API locally with workerd)

LLivog Hey! I'm testing out cloudflare/puppeteer api So I m running this code: ``` pag...

jason•1/29/24, 2:24 PM

I'm not sure where that error is coming from, but it may be coming from within your browser context. Buffer isn't defined in the browser's javascript. Maybe the extractDataextractData method is trying to use it?

LLivog Thanks for the replay, the extractData is simplifying a object to only get the d...

jason•1/29/24, 2:27 PM

Can you provide the stacktrace of the error?

LLivog Never done that, got any link of instructions on how to do that?

jason•1/29/24, 2:30 PM

Add console.log(e.stack)console.log(e.stack) after the "page on response error"

LLivog Hey! I'm testing out cloudflare/puppeteer api So I m running this code: ``` pag...

Chaika•1/29/24, 2:32 PM

I'm not too familiar with puppeter's APIs, but node support with workers is like this:
node_compat=true in your wrangler.toml polyfills some of the nodejs functionality like buffer, ex import { Buffer } from 'buffer'import { Buffer } from 'buffer' would just work
nodejs_compat flag would only expose it via node:buffernode:buffer, have to specifically use it, ex import { Buffer } from 'node:buffer'import { Buffer } from 'node:buffer'.

What version of cloudflare/puppeter are you using? The latest should work with nodejs_compat flag, but before 0.0.4 release it needed node_compat

Denny•1/29/24, 11:49 PM

Hi - Whats the best way to bypass when websites block us from taking a screenshot?

DDenny Hi - Whats the best way to bypass when websites block us from taking a screensho...

jason•1/30/24, 4:01 AM

1) Request the wishes of the website and don't take the screenshot
2) How exactly are they blocking you?

Jjason 1) Request the wishes of the website and don't take the screenshot 2) How exactl...

Denny•1/30/24, 7:11 AM

Cloudflare antibody firewall

DDenny Cloudflare antibody firewall

jason•1/30/24, 8:52 AM

I'm not sure you are going to get many tips about how to circumvent Cloudflare's Firewall on the Cloudflare Discord...

Ssc Hey @Yacine Any chance you put a minimal example in a gist we could look at?

Yacine HmitoOP•1/30/24, 12:46 PM

There's no new code. I just pull the repo and did

npm install

npm install

. I checked that I had a compatible version of node set, but it still didn't work.

jason•2/1/24, 6:47 AM

Can someone explain how this works in CF's distributed network? When a browser is started, is it only in a single location? If I wanted browsers in multiple locations, how would I handle that?

Jjason Can someone explain how this works in CF's distributed network? When a browser i...

Pedro•2/1/24, 9:54 AM

Hey jason! Currently it is started on a single location, we'll be adding support for Geolocation, so that for example the content rendered might change according to the location where connection is initiated

jason•2/1/24, 11:18 AM

Thanks as always @Pedro

jason•2/1/24, 2:07 PM

@Pedro Another question... one of the things that excites me most about this project is the hope of scalability. When I've run puppeteer myself on my servers, the challenge that I've always run into it trying to handle scaling it up. Trying to throttle requests to not start too many processes and run the server out of memory, etc. Is one of the goals of this project actually to be able to solve that?

Is this eventually going to work like other Workers do, where I don't need to care about how many requests I throw at it? Or am I going to have to implement all of the throttling code anyways - but now with the limitation that I can't see what's going on inside the server?

Jjason @Pedro Another question... one of the things that excites me most about this pro...

Pedro•2/1/24, 2:10 PM

Great question! We'll be providing a simple way to reuse sessions and keeping them open, so you can focus on the task at hand. We'll still be setting default limits, mostly to prevent abuse and malicious behaviour, but those can be raised according to you use case

PPedro Great question! We'll be providing a simple way to reuse sessions and keeping th...

jason•2/1/24, 2:13 PM

Thanks. Another way to ask the question. Let's say that I have 1000 urls to crawl. Am I going to need to do the work to manually distribute those requests over multiple sessions, throttle them to not send too many at a time, etc? With a normal worker that doesn't use the browser rendering, I would just POST all 1000 requests as fast as possible to my worker and not worry about it. But I suspect with the browser rendering, I wouldn't be able to do that.

Pedro•2/1/24, 2:19 PM

You can choose to reuse the same session (until expired) to crawls the URLs sequentially or you can choose to create concurrent sessions. It's basically up to you. We're providing an easier way to reuse the sessions, but the "puppeteer" behaviour is still up to you to define, so it's flexible enough to accommodate most use cases. (You can already reuse the same session for multiple URLs using Durable Objects as per the tutorial, we're making it easier)

jason•2/1/24, 2:20 PM

Are you saying that a single session can handle one request at a time? So if I wanted to do 1000 requests in parallel, I would need to start 1000 separate sessions?

Jjason Are you saying that a single session can handle one request at a time? So if I w...

Pedro•2/1/24, 2:25 PM

You can use the same session for multiple tasks like crawling multiple URLs. But if you want to do things in parallel, you just create new concurrent sessions. It's like running a Browser and doing the tasks or Multiple Browsers at the same time and performing multiple task in each one at the same time. Hope I was clear enough

jason•2/1/24, 2:31 PM

Yep, I think I understand.

And I think it means that it's not going to have the worker-level scalability that I hoped it would. At least not without costing a fortune. Thanks for the information!

Jjason Yep, I think I understand. 👍 And I think it means that it's not going to have t...

Pedro•2/1/24, 2:53 PM

Would love to learn more about your use case. Please feel free to ping me and let's make sure we have an offer that also suits more specific and demanding use cases such as yours

SSHAH hello, how do i get off the waiting list for this? i've been on it for ages

jason•2/2/24, 5:21 AM

Did you post in #Browser Rendering API Beta Access ? It’s usually within 24 hours.

kalepail•2/2/24, 7:05 PM

Does this API work in dev? I'm not seeing any indication that it shouldn't but when calling a method that works in prod I'm getting Could not start browser instance. Error: TypeError: Cannot read properties of undefined (reading 'fetch')Could not start browser instance. Error: TypeError: Cannot read properties of undefined (reading 'fetch')

chientrm.com•2/3/24, 10:33 AM

uhm, does this api work for Pages?

Kkalepail Does this API work in dev? I'm not seeing any indication that it shouldn't but w...

jason•2/3/24, 1:53 PM

You need to use

wrangler dev --remote

wrangler dev --remote

Yacine HmitoOP•2/9/24, 2:39 PM

@sc Hello! Did you have the chance to run the Puppeeteer fork locally? Thank you.

Ssc Hi @Yacine Just did and can confirm. Will have to look into it.

Yacine HmitoOP•2/10/24, 8:50 AM

Thank you!

Original message was deleted

Pedro•2/12/24, 8:37 AM

Hey! You can use DOs, regardless of using one session or multiple concurrent sessions. Basically, using DOs allows you to reuse the same session by keeping it open longer, thus if you don't have the need to crawl or execute commands in parallel, one session should be enough. If you have the need to execute commands in parallel, then you can open new "concurrent" sessions

Erisa•2/14/24, 12:54 PM

You can request access in browser-rendering-apiBrowser Rendering API Beta Access

Erisa•2/14/24, 12:54 PM

And you can see in the chat history that it doesnt take too long

Erisa•2/14/24, 12:56 PM

I don't see a message from you in that thread

MyBuddhaMercy•2/22/24, 7:01 AM

Does cloudflare puppeteer support setting proxy?

Prabesh•2/22/24, 7:43 AM

facing the error Dynamic require of "node:stream" is not supported Dynamic require of "node:stream" is not supported when using the legacy syntax with addEventListener.

I pull the repo, run `npm install` with node 18 and it fails with "Cannot use import statement outsi

Similar Threads

Similar Threads

Similar Threads