Firecrawl•16mo ago

API Key for Self Hosted Firecrawl

Hi, I have deployed the self host version of Firecrawl in my system to link it with Dify. In Dify, it requires API Key to integrate. In self host version, we don't have any API Keys or is there a way to generate API Keys. Please help me to solve this issue. Thanks in advance

14 Replies

Adobe.Flash•16mo ago

On the self host you can bypass the need for an API key by disabiling authentication by setting USE_DB_AUTHENTICATION=false in your .env. Then i believe it should just work out of the box in Dify

SaranOP•16mo ago

In Dify, we need to enter API Key and it is a mandatory field I have attached the screenshot for configuring Firecrawl in Dify

Adobe.Flash•16mo ago

Try setting the base url to your local one and on the API key field just writen anything and see if it works. Your localhost should be bypassing the need for the API key so anything you input there should technically work.

grooveseeker•16mo ago

Hey @Adobe.Flash I'm stuck on the same thing as @Saran , I'm certain that the api is up and running correctly cause I get sucessful responses with postman. When I click on "save" on Dify, it stays loading for a long time, and after a while, I see on chrome's dev console that the dify call gets a "status code of 524". I tried debuggin with gpt4turbo and it says:

The response you received is a Cloudflare-generated error page with a status code of 524, which indicates a timeout. This means that the server at cloud.dify.ai did not send a response within the timeout period that Cloudflare expects, causing the request to fail. Here's how you can address and further investigate this issue:

Understanding the Error 524 Response
Error 524 means that Cloudflare was able to make a TCP connection to the origin server, but the origin server did not respond with HTTP headers before the connection timed out. This suggests that the server is reachable but is not processing the request quickly enough.
The error page indicates that the browser and Cloudflare are working correctly, but there's an issue with the host/server where your application is hosted.

The response you received is a Cloudflare-generated error page with a status code of 524, which indicates a timeout. This means that the server at cloud.dify.ai did not send a response within the timeout period that Cloudflare expects, causing the request to fail. Here's how you can address and further investigate this issue:

Understanding the Error 524 Response
Error 524 means that Cloudflare was able to make a TCP connection to the origin server, but the origin server did not respond with HTTP headers before the connection timed out. This suggests that the server is reachable but is not processing the request quickly enough.
The error page indicates that the browser and Cloudflare are working correctly, but there's an issue with the host/server where your application is hosted.

I'm running the local firecrawl with Docker I don't have any error logs from the docker container so, gpt4turbo concluded that the problem is the local instance that is taking to long to send back a reply to Dify, which is weird, can I can send jobs through postman that run flawlessly Also I pulled the latest release of firecrawl, and rebuild the docker image

Adobe.Flash•16mo ago

Hi @grooveseeker , interesting ccing @rafaelmiller here too... I am assuming you have tried what I said above, correct? Just to recap: - USE_DB_AUTHENTICATION=false in your .env - Setting Dify API Key to any text you'd like (fc-abcdefghijk)

grooveseeker•16mo ago

yes, I tried that. I also tried using the same firecrawl api key (just to make sure I followed the same character structure), even tho it shouldn't matter since I have the USE_DB_AUTHENTICATION=false in my .env

Magick•16mo ago

Are you using the python sdk? If yes, you need to specify an api key - https://github.com/mendableai/firecrawl/blob/main/apps/python-sdk/firecrawl/firecrawl.py#L33.

GitHub

firecrawl/apps/python-sdk/firecrawl/firecrawl.py at main · mendable...

🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API. - mendableai/firecrawl

grooveseeker•16mo ago

I wasn't using the python sdk, I was trying to connect Dify to my local instance of firecrawl. Would it be possible to use the sdk with a local instance? hello guys, did any of you find a solution to this?

Adobe.Flash•16mo ago

Hey @grooveseeker we contacted Dify about this. Still haven't heard back We will follow up with them

BrianJM•14mo ago

The same problem exists with Flowise, except Flowise doesn't allow the base url to be adjusted.

Adobe.Flash•14mo ago

@BrianJM Let me reach out to them and see if we can get the base url to be adjusted there!

BrianJM•14mo ago

Thanks @Adobe.Flash. I opened a thread on the Flowise Discord asking if it was possible (maybe I overlooked it).

thousandmiles•14mo ago

hii, have you got dify working with selfhosted firecrawl yet?

C0oola•13mo ago

Hi, I return this parameter when I use the local connection Dify.The local project is already deployed, and I guess it might be an API problem. But the local deployment projects should not need api?Thanks

Gaming

Programming

API Key for Self Hosted Firecrawl

Did you find this page helpful?