F
Firecrawl•15mo ago
andysingal

detect url and query in a prompt?

Hi @Adobe.Flash @rafaelmiller , i have working with Firecrawl. Even though it works nicely for hard coded urls but wanted to take it one step ahead where i can add url and query in the prompt(str) in the Request Body.... Any use cases out there?
7 Replies
andysingal
andysingalOP•15mo ago
Hi @Adobe.Flash @rafaelmiller i was able to detect url and query in a prompt but i get rate limit error for trying to extract? `

message.txt

andysingal
andysingalOP•15mo ago
i can DM the code if you need
rafaelmiller
rafaelmiller•15mo ago
@andysingal the error you sent says that you're reaching to openai rate limits (not firecrawl)
andysingal
andysingalOP•15mo ago
@rafaelmiller can i share the code, may be you tell what seems to be the issue? if thats ok? it works well on simple questions(How many directories in the git repo on https://github.com/mendableai/firecrawl) but error is when i ask "Share Scrapfly code from url https://docs.llamaindex.ai/en/stable/examples/output_parsing/openai_pydantic_program/#extracting-list-of-album-with-parallel-function-calling" Looking forrward to hearing from you
rafaelmiller
rafaelmiller•15mo ago
openai is having some issue right now so that might be it. Also there's a tutorial from openai you can check: https://cookbook.openai.com/examples/how_to_handle_rate_limits
How to handle rate limits | OpenAI Cookbook
andysingal
andysingalOP•15mo ago
@rafaelmiller Thanks but when i run " please share the notebooks within examples folder in the git repo on https://github.com/mendableai/firecrawl" i get : Function call: QueryResult with args: {"query":"please share the notebooks within examples folder in the git repo on","url":"https://github.com/mendableai/firecrawl","result":"The content of the 'examples' folder in the GitHub repository 'mendableai/firecrawl' is not detailed in the provided text. To view or access the notebooks within the 'examples' folder, please visit the repository directly at https://github.com/mendableai/firecrawl/tree/main/examples."} The content of the 'examples' folder in the GitHub repository 'mendableai/firecrawl' is not detailed in the provided text. To view or access the notebooks within the 'examples' folder, please visit the repository directly at https://github.com/mendableai/firecrawl/tree/main/examples.... does Firecrawl not extract that data? here is the code i used 
crawler = FirecrawlApp(api_key= ) try:
            data = crawler.scrape_url(url)
            # relevant_text = data.get("text", "")
        except Exception as e:
            self.logger.error(f"Error while scraping data: {e}")
            raise  # Re-raise the exception to be handled by the caller
crawler = FirecrawlApp(api_key= ) try:
            data = crawler.scrape_url(url)
            # relevant_text = data.get("text", "")
        except Exception as e:
            self.logger.error(f"Error while scraping data: {e}")
            raise  # Re-raise the exception to be handled by the caller
@rafaelmiller @Adobe.Flash do i need to add anything else?
Caleb
Caleb•15mo ago
Hey Andy! I think we need a bit more info to understand the problem. Are you using LLM extract? Or running it trhough llama index in some way. Could you send a snapshot of more of the code?

Did you find this page helpful?