R
Railway•9mo ago
sickjoker

Celery connection issue

Hi! Just recently I've been having issues with my celery instance. It was working all this time. It can't connect to redis (everything is dockerized). Here is the following issue: Cannot connect to {redacted redis address} Lookup timed out.. - Has anyone had the same issue?
64 Replies
Percy
Percy•9mo ago
Project ID: N/A
sickjoker
sickjoker•9mo ago
N/A
Brody
Brody•9mo ago
the redis address would be very important to help us debug this
sickjoker
sickjoker•9mo ago
redis-15409.c281.us-east-1-2.ec2.cloud.redislabs.com:15409 🙂
Brody
Brody•9mo ago
oh so not on railway is there some kind of whitelist you need to do?
sickjoker
sickjoker•9mo ago
well I also tried railway as well I'm having the same issue no matter the redis address however it works just fine when connecting locally
Brody
Brody•9mo ago
can we focus on getting you connected to a railway hosted redis database?
sickjoker
sickjoker•9mo ago
local instance to production redis that's dine with me fine*
Brody
Brody•9mo ago
works better for me since i know nothing about redislabs
sickjoker
sickjoker•9mo ago
monorail.proxy.rlwy.net:34378
Brody
Brody•9mo ago
can i see a screenshot of your railway project please?
sickjoker
sickjoker•9mo ago
sure give me a sec, let me configure redis on railway
Brody
Brody•9mo ago
if redis isn't configured, where did you get that domain from?
sickjoker
sickjoker•9mo ago
so that domain is from a heroku instance, I'm having the same issue and decided to try railway to see if it was a isolated instance My application had been working just fine up until today
Brody
Brody•9mo ago
im talking about this domain
sickjoker
sickjoker•9mo ago
Oh I had tested a setup earlier I just tried connecting to it and i do get the same error [2024-02-26 17:31:11,560: ERROR/MainProcess] consumer: Cannot connect to redis://default:**@monorail.proxy.rlwy.net:34378/0: Error -3 connecting to monorail.proxy.rlwy.net:34378. Lookup timed out..
Brody
Brody•9mo ago
please show me the screenshot of your project
sickjoker
sickjoker•9mo ago
just a normal screenshot? or is there a railway function sorry, new to this
sickjoker
sickjoker•9mo ago
No description
sickjoker
sickjoker•9mo ago
ignore the flask
Brody
Brody•9mo ago
okay can show me how you are trying to connect to it
sickjoker
sickjoker•9mo ago
let me make a quick loom
sickjoker
sickjoker•9mo ago
Thanks for the help! I truly appreciate it
Brody
Brody•9mo ago
very helpful video, what do you have REDIS_ENDPOINT and REDIS_PORT set to in your service variables?
sickjoker
sickjoker•9mo ago
give me one sec REDIS_ENDPOINT=default:**@monorail.proxy.rlwy.net REDIS_PORT=34378 asterisk is the password btw
Brody
Brody•9mo ago
instead of trying to build the url yourself, just use a REDIS_URL variable in the celery config file, then in your service variables set REDIS_URL to ${{Redis.REDIS_PRIVATE_URL}} this is unlikely to fix the main issue, but its best we do this instead
sickjoker
sickjoker•9mo ago
okay got it - would my config look like this then? import os from dotenv import load_dotenv load_dotenv() REDIS_URL = os.environ.get('REDIS_URL', '6379') # default to 6379 if not provided imports = ["tasks"] broker_url = REDIS_URL result_backend = REDIS_URL task_serializer = 'json' result_serializer = 'json' accept_content = ['json'] timezone = 'America/Chicago' enable_utc = True broker_connection_retry_on_startup = True
Brody
Brody•9mo ago
lets omit the dotenv for now, since it looks like you are committing that file to your repo
sickjoker
sickjoker•9mo ago
okay gotcha
Brody
Brody•9mo ago
import os

imports = ["tasks"]

broker_url = os.environ['REDIS_URL']
result_backend = os.environ['REDIS_URL']
task_serializer = 'json'
result_serializer = 'json'
accept_content = ['json']
timezone = 'America/Chicago'
enable_utc = True
broker_connection_retry_on_startup = True
import os

imports = ["tasks"]

broker_url = os.environ['REDIS_URL']
result_backend = os.environ['REDIS_URL']
task_serializer = 'json'
result_serializer = 'json'
accept_content = ['json']
timezone = 'America/Chicago'
enable_utc = True
broker_connection_retry_on_startup = True
sickjoker
sickjoker•9mo ago
oaky got it - railway gives me two options REDIS_URL and REDIS_PRIVATE_URL when I'm setting env varible
Brody
Brody•9mo ago
please reference my message
sickjoker
sickjoker•9mo ago
okay got it, deploying now okay deployed -s ame issue
Brody
Brody•9mo ago
show me the new error please
sickjoker
sickjoker•9mo ago
[2024-02-26 17:54:49,592: ERROR/MainProcess] consumer: Cannot connect to redis://default:**@redis.railway.internal:6379//: Error -3 connecting to redis.railway.internal:6379. Lookup timed out..
Brody
Brody•9mo ago
cool, send your dockerfile please, and going forward please enclose logs and code (or similar) in code blocks
sickjoker
sickjoker•9mo ago
sorry haha , how do i enclose in code blocks
Brody
Brody•9mo ago
triple backticks
sickjoker
sickjoker•9mo ago
Brody
Brody•9mo ago
try this instead
# Dockerfile.worker

# Use an official Python runtime as a parent image
FROM python:3.11.1

WORKDIR /app

COPY requirements.txt .
RUN pip install -r requirements.txt

COPY . .

# Command to run the Celery worker
CMD sleep 3 && celery -A tasks worker --loglevel=info --pool=eventlet
# Dockerfile.worker

# Use an official Python runtime as a parent image
FROM python:3.11.1

WORKDIR /app

COPY requirements.txt .
RUN pip install -r requirements.txt

COPY . .

# Command to run the Celery worker
CMD sleep 3 && celery -A tasks worker --loglevel=info --pool=eventlet
sickjoker
sickjoker•9mo ago
trying same issue
Brody
Brody•9mo ago
interesting
sickjoker
sickjoker•9mo ago
yeah, I'm not sure if it's a rediscloud issue but status have been okay with a few urgent fixes that happend last two days as I said before, everything worked properly until today does railway use the rediscloud service?
Brody
Brody•9mo ago
railway deploys a redis docker image
sickjoker
sickjoker•9mo ago
hmm. yeah, it's really odd driving me insane
Brody
Brody•9mo ago
increase the sleep to 10 seconds?
sickjoker
sickjoker•9mo ago
sure ill try it same issue :/
Brody
Brody•9mo ago
im kinda out of ideas, you sure you are usng the dockerfile i provided?
sickjoker
sickjoker•9mo ago
Yes I am :9 😦 I'm dumbfounded as well. It literally worked for months before today
Brody
Brody•9mo ago
and you haven't modified anything at all about the redis database on railway right? absolutely nothing at all?
sickjoker
sickjoker•9mo ago
Nope nothing I even tried rabbitmq instead of redis same issue
Brody
Brody•9mo ago
let me do some thinking, could you share your repo, or add me to it?
sickjoker
sickjoker•9mo ago
Thank you! This is under a client repo but I can create another and add you in
Brody
Brody•9mo ago
yes please brody192
sickjoker
sickjoker•9mo ago
Hmm well, here's another thing I tried a skeleton celery and it's connecting I'm going to look into this further I'll keep you posted 🙂 Thank you Brody I will definitley buy you some coffee
Brody
Brody•9mo ago
theres no need but i very much appreciate the gesture
sickjoker
sickjoker•9mo ago
Here you go ! https://www.buymeacoffee.com/brody192/c/8635280 - The world needs mroe people like you. Much appreciated
Buy Me a Coffee
Brody is Helping in the Railway Community Discord server!
Hello! 👋I don't work for Railway 🫠
sickjoker
sickjoker•9mo ago
I figured out what it was - it was the concurrency pool. Eventlet doesn't want to play nice with Docker I guess
Brody
Brody•9mo ago
sorry i couldn’t solve this but happy you have! and thank you so much for the train!! a sleep is still needed, but you can bump the sleep down to 3 seconds
sickjoker
sickjoker•9mo ago
Okay sounds good! Why do you recommend the sleep? You still helped, I appreciate it
Brody
Brody•9mo ago
because the private network takes about 3 seconds to be able to respond to dns lookups
sickjoker
sickjoker•9mo ago
Ah okay gotcha makes sense thank you again
Brody
Brody•9mo ago
happy to help where i can!
Want results from more Discord servers?
Add your server