ideal input for `search` param in /map endpoint
hi @rafaelmiller @Caleb i was testing out the /map endpoint to fetch a given specific subset of links for a company's domain.
so i wanted to understand what would an ideal query to the
search param should look like ?
for eg. if i want a company's global offices information i would ideally go to the page where locations , offices information is there, soo in the search query i passed location keyword as the param value.
I got the website where the locations are present however i got additional > 50 websites which were not required at all.
so i am hoping to narrow down the filtered list ..
i mean one potential solution i can think of is adding an additional llm call on top of these filtered queries to further filter the specific web pages but i believe it will add an overhead in the service time ?
another thing i noticed was given a url and search param value, if we run multiple iterations the total number of results is varied across iterations ( even though url and search param value is same ). This inconsistency might be an issue for enterprise solutions.
the frequency of this is less and i am sure u guys might have thought of this or considered but still any visibility would surely help !
potential solution that i can think of is maybe implement a caching mechanism with a specified ttl value ? - update : i noticed u guys have already implemented this .. but what if i trigger the same query after the ttl ? will i get the same number of results ?
sharing this sheet for some reference : https://docs.google.com/spreadsheets/d/1EmIxn6sVOheugBu1RASAadJab5C0OqysLmaYjTpb-i8/edit?usp=sharingGoogle Docs
firecrawl testing
3 Replies
Hi @lazycoderai ! Great question, ccing @Adobe.Flash here.
Hey! This is a great question.
Right now, we just return as many results as we can find. In the short term, you may want to pick only the first 10 results for example. Combinded with your solution, this would probably work well, even if not perfect.
In the long term, we should probably add a relevancy threshold @Adobe.Flash
On the second point, I’m not sure why you’re getting different results, we will look into it!
Thanks @Caleb for the response.