Crawl not working on self-hosted with external redis (Upstash)

The bull dashboard doesn't seem to be showing the web scraper queue when it is connected to Upstash redis. I did not do anything like IP whitelisting etc that should block the connection.

I also got no concerning log messages

api-1                 | Web scraper queue created
api-1                 | Worker 38 started
api-1                 | Worker 38 listening on port 3002
api-1                 | For the UI, open http://0.0.0.0:3002/admin/@/queues
api-1                 | 
api-1                 | 1. Make sure Redis is running on port 6379 by default
api-1                 | 2. If you want to run nango, make sure you do port forwarding in 3002 using ngrok http 3002 
api-1                 | Connected to Redis Session Store!

api-1                 | Web scraper queue created
api-1                 | Worker 38 started
api-1                 | Worker 38 listening on port 3002
api-1                 | For the UI, open http://0.0.0.0:3002/admin/@/queues
api-1                 | 
api-1                 | 1. Make sure Redis is running on port 6379 by default
api-1                 | 2. If you want to run nango, make sure you do port forwarding in 3002 using ngrok http 3002 
api-1                 | Connected to Redis Session Store!

api-1                 | Web scraper queue created
api-1                 | Worker 38 started
api-1                 | Worker 38 listening on port 3002
api-1                 | For the UI, open http://0.0.0.0:3002/admin/@/queues
api-1                 | 
api-1                 | 1. Make sure Redis is running on port 6379 by default
api-1                 | 2. If you want to run nango, make sure you do port forwarding in 3002 using ngrok http 3002 
api-1                 | Connected to Redis Session Store!

api-1                 | Web scraper queue created
api-1                 | Worker 38 started
api-1                 | Worker 38 listening on port 3002
api-1                 | For the UI, open http://0.0.0.0:3002/admin/@/queues
api-1                 | 
api-1                 | 1. Make sure Redis is running on port 6379 by default
api-1                 | 2. If you want to run nango, make sure you do port forwarding in 3002 using ngrok http 3002 
api-1                 | Connected to Redis Session Store!

It also seems like the redis connection is successful. It might be worth noting that using local docker composed redis works.

However, whenever I try to do crawl job, I got socket hangup instead of jobId. Am I missing something here? I attached the IP address of the service in the image and also attached the postman request for crawl.

Is there any missing steps to connect to external redis providers like Upstash?

purnamaOP•6/15/24, 10:59 PM

Also, great job team @

crawl

Adobe.Flash•6/16/24, 4:54 AM

Thank you @purnama! That's quite odd. Try manually running pnpm run startpnpm run start and pnpm run workerspnpm run workers seperately and see if that helps

Adobe.Flash•6/16/24, 4:55 AM

Essentially they should be 2 different processes and it seems like for some reason here only

start

start

is getting initiated

purnamaOP•6/16/24, 3:57 PM

I see, I will try again!

purnamaOP•6/16/24, 6:37 PM

@Adobe.Flash I tried, it doesn't work, there's no suspicious logs either. Any pointers on why this is the case?

purnamaOP•6/19/24, 8:52 PM

Hi Team, bumping this up!

Adobe.Flash•6/20/24, 5:38 PM

Hey @purnama not really sure, ccing @rafaelmiller here who has dealt more closely with the self hosting issues.

But my guess is that the workers arent being initialized correctly. Also, make sure you have a redis instance running!

rafaelmiller•6/20/24, 5:46 PM

hey @purnama ! It seems like your server isn't connecting properly with redis (though the logs suggest it does). Have you configured the

.env

.env

variable for your external redis (upstash)? It’s possible that the credentials are missing from the

.env

.env

variable (I'm not familiar with Upstash, so it's just a guess)

purnamaOP•6/20/24, 5:51 PM

Hi @rafaelmiller @Adobe.Flash thanks so much for following up!

I can share the redis string through DM so you can try to quickly spin one up on your end?

I have tried:

Upstash
DigitalOcean Redis (I deploy firecrawl on DO)

Also I'd be happy to share how I deploy firecrawl on DO for yall if that's something useful

Crawl not working on self-hosted with external redis (Upstash)

Similar Threads

Similar Threads

Similar Threads