F
Firecrawl14mo ago
Andrew

What is the best way to dedupe URLs with the same content?

For example, https://supabase.com/docs/reference/javascript/introduction https://supabase.com/docs/reference/javascript/installing which I get from /map (attached image) The reason I get the same content is because although the URL is different, it is actually the same page
JavaScript: Introduction | Supabase Docs
Supabase API reference for JavaScript: Introduction
JavaScript: Installing | Supabase Docs
Supabase API reference for JavaScript: Installing
No description
2 Replies
mogery
mogery13mo ago
Hi there @Andrew! Currently, /map is not aware of the page content -- however, we plan to change this in the future.

Did you find this page helpful?