Crawlee & Apify•3y ago

Need help compiling crawlee in react

Hi everyone! I'm trying to integrate the crawlee library inside my react app for a social scrape project, but as soon as I import the PlaywrightCrawler module I get the following compile error:

ERROR in ./node_modules/@crawlee/browser/internals/browser-crawler.js
Module build failed (from ./node_modules/react-scripts/node_modules/babel-loader/lib/index.js):
TypeError: /Applications/MAMP/htdocs/scouting/client/node_modules/@crawlee/browser/internals/browser-crawler.js: Cannot read properties of null (reading 'type')

ERROR in ./node_modules/@crawlee/browser/internals/browser-crawler.js
Module build failed (from ./node_modules/react-scripts/node_modules/babel-loader/lib/index.js):
TypeError: /Applications/MAMP/htdocs/scouting/client/node_modules/@crawlee/browser/internals/browser-crawler.js: Cannot read properties of null (reading 'type')

Along with numerous warnings, all of the type:

WARNING in ./node_modules/proxy-chain/dist/forward.js
Module Warning (from ./node_modules/source-map-loader/dist/cjs.js):
Failed to parse source map from '/Applications/MAMP/htdocs/scouting/client/node_modules/proxy-chain/src/forward.ts' file: Error: ENOENT: no such file or directory

WARNING in ./node_modules/proxy-chain/dist/forward.js
Module Warning (from ./node_modules/source-map-loader/dist/cjs.js):
Failed to parse source map from '/Applications/MAMP/htdocs/scouting/client/node_modules/proxy-chain/src/forward.ts' file: Error: ENOENT: no such file or directory

I am using react 18.2.0 and node 16.19.1 version (>= 16, as specified by the documentation), for clarity here is also my package.json. I already tried deleting /node_modules and package-lock.json and re-running npm install, but the problem persists. Anyone know how to fix? Thx! package.json:

{
  "dependencies": {
    "@craco/craco": "7.1.0",
    "@hookform/resolvers": "3.0.0",
    "@reduxjs/toolkit": "^1.2.5",
    "autoprefixer": "10.4.5",
    "bootstrap": "5.2.3",
    "bs-stepper": "1.7.0",
    "classnames": "2.3.1",
    "craco": "0.0.3",
    "crawlee": "^3.0.1",
    "fs": "^0.0.1-security",
    "node-polyfill-webpack-plugin": "^2.0.1",
    "prismjs": "1.19.0",
    "prop-types": "15.7.2",
    "react": "18.2.0",
    "react-dom": "18.2.0",
    "react-hook-form": "7.43.7",
    "react-redux": "8.0.5",
    "react-router-dom": "6.2.2",
    "react-scripts": "5.0.1",
    "reactstrap": "9.0.1",
    "redux": "4.0.5",
    "sass": "1.32.13",
    "styled-components": "5.1.1",
    "yarn": "1.21.1",
    "yup": "1.0.2"
  },
  "scripts": {
    "start": "craco start",
    "build": "craco build",
  }
}

{
  "dependencies": {
    "@craco/craco": "7.1.0",
    "@hookform/resolvers": "3.0.0",
    "@reduxjs/toolkit": "^1.2.5",
    "autoprefixer": "10.4.5",
    "bootstrap": "5.2.3",
    "bs-stepper": "1.7.0",
    "classnames": "2.3.1",
    "craco": "0.0.3",
    "crawlee": "^3.0.1",
    "fs": "^0.0.1-security",
    "node-polyfill-webpack-plugin": "^2.0.1",
    "prismjs": "1.19.0",
    "prop-types": "15.7.2",
    "react": "18.2.0",
    "react-dom": "18.2.0",
    "react-hook-form": "7.43.7",
    "react-redux": "8.0.5",
    "react-router-dom": "6.2.2",
    "react-scripts": "5.0.1",
    "reactstrap": "9.0.1",
    "redux": "4.0.5",
    "sass": "1.32.13",
    "styled-components": "5.1.1",
    "yarn": "1.21.1",
    "yup": "1.0.2"
  },
  "scripts": {
    "start": "craco start",
    "build": "craco build",
  }
}

7 Replies

dependent-tan•3y ago

I want to clarify what exactly you are trying to do. Do you want to run crawlee (node.js env) in a browser app?

typical-coralOP•3y ago

yes, the idea is to run crawlee within web management. the goal is to scrape some social profiles and allow the user to solve captchas when they arise. Am I wrong in the approach? *web management app

dependent-tan•3y ago

You would need a node js server that runs your crawler, and make api calls to it

typical-coralOP•3y ago

I understand, and how can I solve captchas with this? many thanks in advance

dependent-tan•3y ago

The best case is not getting it at all, it's a bit tedious task to solve captcha. There are paid services that provide api and integration libs. You can also find some open source libraries that aim to solve some types of captcha.

ambitious-aqua•3y ago

Its really not expected approach, actors designed to be executed either in Apify cloud or under your own nodejs server, in turn web apps expected to do API calls either to Apify or your server.

typical-coralOP•3y ago

I switched to this setup successfully, thanks for your support

Gaming

Programming

Need help compiling crawlee in react

Did you find this page helpful?