Unable to use Crawlee on AWS Lambda: hile loading shared libraries: libnss3.so: cannot open shared o
I have a problems when I deploy with CloudFormation on AWS Lambda.
NODE versione: 16
Crawlee: 3.3
"aws-cdk-lib": "2.29.1",
"aws-sdk": "^2.1163.0",
"chrome-aws-lambda": "10.1.0",
"commander": "^9.3.0",
"constructs": "10.1.42",
"crawlee": "^3.3.0",
"@sparticuz/chromium":"112.0.2",
"lodash": "^4.17.21",
"playwright": "^1.27.1",
CODE:
import { PlaywrightCrawler } from 'crawlee';
const awsChromium = require('chrome-aws-lambda');
const crawler = new PlaywrightCrawler({
launchContext:{
launchOptions: {
executablePath: await awsChromium.executablePath || ""
}
},
useSessionPool: true,
sessionPoolOptions: { maxPoolSize: 100 },
persistCookiesPerSession:true,
maxRequestsPerMinute: 10,
async requestHandler({ request, page, enqueueLinks, log, infiniteScroll, slowMo }) {
const title = await page.title();
console.log("Crawler-full CrawlPage:" + JSON.stringify(configurationCrawl));
.......
await crawler.run(["https://www.google.com"]);
On local all it's ok, but when I deploy I have this error:
<launching> /tmp/chromium --disable-field-trial-config --disable-background-networking --enable-
...
dir=/tmp/playwright_chromiumdev_profile-VU4eqQ --remote-debugging-pipe about:blank
<launched> pid=26
[pid=26][err] /tmp/chromium: error while loading shared libraries: libnss3.so: cannot open shared object file: No such file or directory
[pid=26] <process did exit: exitCode=127, signal=null>
[pid=26] starting temporary directories cleanup
I see googling say it downgrading node version at 14 can solve, but I can because Crawlee give me error for Node 14 version.
Have you some ideas about it?
7 Replies
fair-rose•3y ago
It seems like the problem is not with crawlee itself, but with the cromium. As for the node version - It's not possible to run it on Node 14, it requires 15+ or 16+. I don't have experience with AWS, but do you need this specific version?
harsh-harlequinOP•3y ago
I think it's crawlee because only puppeteer don't have this error.
I can choose node 14, 16 in lambda function but I haven't care about it specific version
fair-rose•3y ago
But still - any chance you could try it with regular puppeteer? It could be coming from crawlee indirectly. But in any case - I would pass it to the team, but I know for sure that there were some user running it on AWS...
harsh-harlequinOP•3y ago
Do you know about some user on this groups will achieved that (running crawlee on AWS Lambda? @Andrey Bykov
fair-rose•3y ago
Not from the top of my head. Which docker image are you using? One of our devs suspect it's just some missing lib in docker image.
harsh-harlequinOP•3y ago
Nope I upload code with cloud formation and not use docker image. I find the problem was I use with node 16 chrome-aws-lambda it not supported by.
Now it work with a new supported library a fork of chrome-aws-lambda
fair-rose•3y ago
Got it - great that it worked out in the end 👌