crawlUrlAndWatch working on local but fails on Production
I am using crawlUrlAndWatch to crawl through domains, it works perfectly fine in my local system but it fails in production. The error I get is:
SyntaxError: The URL's protocol must be one of "ws:", "wss:", or "ws+unix:"
The frustrating part is that the crawl gets registered in my activity and platform charges the credits for that action. It's breaking my production and charging me for it. Any solutions please?
The example website can be: https://www.callprep.io (I have tried removing all different combinations to remove have https and all but it remains the same.)
I am using Next Js 14.2.14, my production instance is in AWS private subnet and has an attached NAT gateway for internet access.
Please help me with this issue!
Here is the code-snippet I am using (some components have been removed for privacy):
export async function crawlWebsite(
url: string,
...
) {
const job_id = v4();
const app = new FirecrawlApp({ apiKey: process.env.NEXT_FIRECRAWL_API_KEY });
const watch = await app.crawlUrlAndWatch(url, { excludePaths: ['blog/*'], limit: 500 });
watch.addEventListener("document", doc => {
if (doc.detail.markdown) {
handleDocumentCrawl(
doc.detail,
...
);
}
});
watch.addEventListener("error", err => {
console.error("ERR", err.detail.error);
});
watch.addEventListener("done", state => {
console.log("DONE", state.detail.status);
// Wait for 2 minutes (120000 milliseconds) after Done signal
setTimeout(() => {
console.log("Ending watch after 2 minutes");
watch.close();
}, 120000);
});
}
Callprep 2
CallPrep | AI-powered sales automation
CallPrep is an all-in-one AI-powered sales copilot designed for Account Executives in B2B companies. Streamline your sales process with real-time insights, automated follow-ups, and CRM integration. Boost win rates, shorten sales cycles, and close more deals with advanced automation and deal coaching features. Book a demo today!
0 Replies