CA
Crawlee & Apify•2y ago
rare-sapphire

Module not found in NextJs projects

I am trying to run a playwright crawler with Crawlee and for some reason I'm getting a error saying the puppeteer module cannot be found. I've installed npm install crawlee playwright but not puppeteer Do i need puppeteer also to run playwright? i dont think so... 
https://nextjs.org/docs/messages/module-not-found

Import trace for requested module:
./node_modules/@crawlee/puppeteer/index.js
./node_modules/crawlee/index.js
./node_modules/crawlee/index.mjs
./app/actions/crawlee.ts
./app/api/local/crawlee/route.ts
 ⨯ ./node_modules/@crawlee/puppeteer/internals/utils/puppeteer_utils.js:190:21
Module not found: Can't resolve 'puppeteer/package.json'
https://nextjs.org/docs/messages/module-not-found

Import trace for requested module:
./node_modules/@crawlee/puppeteer/index.js
./node_modules/crawlee/index.js
./node_modules/crawlee/index.mjs
./app/actions/crawlee.ts
./app/api/local/crawlee/route.ts
 ⨯ ./node_modules/@crawlee/puppeteer/internals/utils/puppeteer_utils.js:190:21
Module not found: Can't resolve 'puppeteer/package.json'
No description
4 Replies
rare-sapphire
rare-sapphireOP•2y ago
SOLUTION FOUND: For nextjs projects, add 'crawlee' and 'playwright' to the webpack externals array like this
No description
rare-sapphire
rare-sapphireOP•2y ago
webpack: (config, { isServer }) => {
    if (isServer) {
      config.externals.push(
        'crawlee',
        'puppeteer',
        'playwright',
        // ... any other server-side only packages
      );
    }
    return config;
  },
webpack: (config, { isServer }) => {
    if (isServer) {
      config.externals.push(
        'crawlee',
        'puppeteer',
        'playwright',
        // ... any other server-side only packages
      );
    }
    return config;
  },
optimistic-gold
optimistic-gold•17mo ago
Hello @tim were you able to make this work in an Edge function? I get Error [ReferenceError]: crawlee is not defined when in edge functions. OK in normal serverless API routes. Actually when deployed it doesn't work in normal serverless functions. I get [cause]: Error: Cannot find module 'playwright'. @tim were you able to make playwright run in vercel serverless functions at all?
extended-salmon
extended-salmon•2mo ago
Hi, I get this error: Failed to compile ./node_modules/.pnpm/@[email protected][email protected]/node_modules/@crawlee/puppeteer/internals/utils/puppeteer_utils.js:224:22 Module not found: Can't resolve 'puppeteer/package.json' 222 | return client.send(command, ...args); 223 | }
224 | const jsonPath = require.resolve('puppeteer/package.json');
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 225 | const parsed = JSON.parse(await (0, promises_1.readFile)(jsonPath, 'utf-8')); 226 | throw new Error(Cannot detect CDP client for Puppeteer ${parsed.version}. You should report this to Crawlee, mentioning the puppeteer version you are using.); 227 | } https://nextjs.org/docs/messages/module-not-found
Module Not Found

Did you find this page helpful?