Is the url pattern, includes, excludes, using regular expression?
Hi new to firecrawl, anyone know if the url pattern, includes, excludes, using regular expression?
4 Replies
We don't support regular expressions there at the moment, just glob patterns like /*
What do you need regex for in particular?
Hi @Caleb ,
I'm also encountering an issue with another website: https://ellijewelry.com/products/ohrstecker-2er-set-basic-kristall-weiss-925er-sterling-silber?variant=45406085185819. I want to scrape all URLs that include the variant part, but using product/* isn't capturing those URLs. Specifically, variant=* is not working as expected.
Is there a roadmap to include regex in the URL include feature? Or does anyone have any suggestions on how to effectively capture these URLs?
Got it. Is the issue that you're not getting urls with the variant=<id> pattern? Or that you can't set te includesPaths to ONLY get links with that issue
This may not be a issue with the filtering, but an issue with getting those urls in the first place
Yes I am not getting urls with variant=<id> pattern. Then I tried to use the tag variant= pattern, but it returned null as it could not match it. @Caleb