R
Railway11mo ago
hkarthik

What I thought was a simple python upgrade broke my deployed flask service. Need some help.

Project ID: b06af9f3-d9c9-4d4c-a74f-c5521f149984/service/872870dd-f98a-486c-9f14-09fcd5bc43e5
87 Replies
Percy
Percy11mo ago
Project ID: b06af9f3-d9c9-4d4c-a74f-c5521f149984,872870dd-f98a-486c-9f14-09fcd5bc43e5
Brody
Brody11mo ago
well what broke? show us some logs please?
hkarthik
hkarthik11mo ago
I'm using a lightweight task library called Huey, which uses Redis as the storage layer. That's failing to bootstrap in my init. File "/app/myapp/init.py", line 42, in create_app from . import tasks File "/app/myapp/tasks/init.py", line 2, in <module> from .cli import register_cli_blueprint File "/app/myapp/tasks/cli.py", line 6, in <module> from .tasker import TASKER File "/app/myapp/tasks/tasker.py", line 30, in <module> TASKER: Tasker = Tasker(current_app) ^^^^^^^^^^^^^^^^^^^ File "/app/myapp/tasks/tasker.py", line 10, in init self.init_app(app) File "/app/myapp/tasks/tasker.py", line 14, in init_app huey_instance = RedisHuey( ^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/huey/api.py", line 116, in init self.storage = self.create_storage() ^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/huey/api.py", line 135, in create_storage return self.get_storage(self.storage_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/huey/api.py", line 147, in get_storage return Storage(self.name, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/huey/storage.py", line 369, in init raise ConfigurationError('"redis" python module not found, cannot ' huey.exceptions.ConfigurationError: "redis" python module not found, cannot use Redis storage backend. Run "pip install redis" to
Brody
Brody11mo ago
please provide logs as .txt or .log files or at minimum wrap that text in triple backticks but regardless, do you have redis in your requirements.txt file?
hkarthik
hkarthik11mo ago
sorry, is there an easy way to do that from Railway's UI? I'm using a Pipfile
Brody
Brody11mo ago
do you have a requirements.txt file though?
hkarthik
hkarthik11mo ago
and yes, Redis is specified there no I do not
Brody
Brody11mo ago
https://bookmarklets.up.railway.app/log-downloader/ use that to send both your build and deploy logs please
Brody
Brody11mo ago
why do you have redis in the setup phase? what have you done there?
hkarthik
hkarthik11mo ago
ah, I added that in a fit of debugging and trying random things last night it was not there when this was all working on Python 3.8
Brody
Brody11mo ago
I assume you are using a nixpacks.toml file? if so, send that over please
hkarthik
hkarthik11mo ago
I overwrote the start command for this service (my repo is a mono repo) I have it set as
flask run-huey
flask run-huey
inside of Railway's UI to override
Brody
Brody11mo ago
who wrote that file? it looks like something chatgpt would write after scanning the nixpacks docs site
hkarthik
hkarthik11mo ago
initially generated but I made a few modifications. Had to debug the dockerfile locally to get to this point if there are ways to simplify it, I'm all ears
Brody
Brody11mo ago
was it chatgpt generated, I'm curious
hkarthik
hkarthik11mo ago
I don't think so, I might have copied it from another repo but like I said, I made some modifications after lots of local debugging with Docker
Brody
Brody11mo ago
can you share your repo?
hkarthik
hkarthik11mo ago
sure, send me your GH username
Brody
Brody11mo ago
brody192
hkarthik
hkarthik11mo ago
done, repo is called kdp-analyzer
Brody
Brody11mo ago
so i can assume that you do have the root directory in the service settings set to /backend right?
hkarthik
hkarthik11mo ago
yes
Brody
Brody11mo ago
either way, delete your Procfile and nixpacks.toml file from the repo and locally, we will then go from there
hkarthik
hkarthik11mo ago
okay, let me do that and push a commit pushed
Brody
Brody11mo ago
what version of python do you use locally
hkarthik
hkarthik11mo ago
3.11 3.11.5 to be exact
Brody
Brody11mo ago
add a .python-version with 3.11 in it
hkarthik
hkarthik11mo ago
okay, and I assume make sure that's not ignored and goes into git?
Brody
Brody11mo ago
it tells nixpacks to use python 3.11 otherwise the default is 3.8
hkarthik
hkarthik11mo ago
oh got it ok pushed that
Brody
Brody11mo ago
youll now want to remove any build or start commands you may have set in the service settings, and add this railway.json file to your backend folder
{
"$schema": "https://schema.up.railway.app/railway.schema.json",
"build": {
"builder": "NIXPACKS"
},
"deploy": {
"startCommand": "gunicorn wsgi:gunicorn_app"
}
}
{
"$schema": "https://schema.up.railway.app/railway.schema.json",
"build": {
"builder": "NIXPACKS"
},
"deploy": {
"startCommand": "gunicorn wsgi:gunicorn_app"
}
}
you where using the flask command to run your app, as far as im aware, that starts a development server
hkarthik
hkarthik11mo ago
right that's the web server
Brody
Brody11mo ago
we do not want to run the server in development mode on railway
hkarthik
hkarthik11mo ago
the flask command was used for only the task manager huey the web server was run with
gunicorn wsgi:gunicorn_app --log-level=debug
gunicorn wsgi:gunicorn_app --log-level=debug
Brody
Brody11mo ago
did you have two services for the backend or something?
hkarthik
hkarthik11mo ago
yeah originally I had one the command was set to "gunicorn wsgi:gunicorn_app --log-level=debug & flask run-huey"
Brody
Brody11mo ago
not ideal
hkarthik
hkarthik11mo ago
I think I have to go back to that, unfortunately both use a shared file system files get uploaded via web, and processed via Huey I was going to move to an S3 bucket that could be shared at some point is it better to have two separate services or one?
Brody
Brody11mo ago
its not ideal because that will run gunicorn in the background, railway wont be able to detect a crash
hkarthik
hkarthik11mo ago
right
Brody
Brody11mo ago
you said you need a shared fs, so you need to do one service until railway supports attaching a volume to multiple services
hkarthik
hkarthik11mo ago
exactly
Brody
Brody11mo ago
to run two things in one service we need to use a tool called parallel but we will come back to that later, lets get the web server part of this working first so remove one of the backend services
hkarthik
hkarthik11mo ago
I'm trying, the railway UI appears broken might have to use the CLI
Brody
Brody11mo ago
broken?
hkarthik
hkarthik11mo ago
No description
hkarthik
hkarthik11mo ago
Delete button won't become enabled
Brody
Brody11mo ago
haha yes i know, add a space to the begging of that input box
hkarthik
hkarthik11mo ago
omg ok got it
Brody
Brody11mo ago
your service name has a space at the begging too, this isnt a bug
Brody
Brody11mo ago
see the extra underscore
No description
hkarthik
hkarthik11mo ago
okay re-pushing now interesting, wonder how that got there ah, now we're back to where we were
Brody
Brody11mo ago
okay lets try something, make a requirements.txt file
flask
flask-sqlalchemy
sqlalchemy
psycopg2-binary
pandas
openpyxl
babel
forex-python
flask-caching
python-dotenv
click
huey
redis
gunicorn
flask-migrate
flask
flask-sqlalchemy
sqlalchemy
psycopg2-binary
pandas
openpyxl
babel
forex-python
flask-caching
python-dotenv
click
huey
redis
gunicorn
flask-migrate
hkarthik
hkarthik11mo ago
ok trying that ok we got past that error, something different now likely related to missing ENV vars for the SqlAlchemy DB connection one interesting thing flashed in deploy logs that might be the culprit, one second
Brody
Brody11mo ago
is your database hosted on railway in the same project?
hkarthik
hkarthik11mo ago
yes
#10 13.63 DEPRECATION: huey is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559

#10 13.63 Running setup.py install for huey: started



#10 14.22 Running setup.py install for huey: finished with status 'done'
#10 13.63 DEPRECATION: huey is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559

#10 13.63 Running setup.py install for huey: started



#10 14.22 Running setup.py install for huey: finished with status 'done'
Brody
Brody11mo ago
dont worry about that right now show me a screenshot of your service variables please
hkarthik
hkarthik11mo ago
No description
Brody
Brody11mo ago
use variable references that arent shared please
hkarthik
hkarthik11mo ago
okay, trying to unshare
hkarthik
hkarthik11mo ago
nice, looks like its all back up
Brody
Brody11mo ago
for a sanity check, mind sharing another service variables screenshot?
hkarthik
hkarthik11mo ago
No description
hkarthik
hkarthik11mo ago
do you suspect the shared variables weren't working right?
Brody
Brody11mo ago
you can remove the port variable btw okay so now everything works, right?
hkarthik
hkarthik11mo ago
yes I'm still confused about the requirements.txt and why that's needed
Brody
Brody11mo ago
no clue
hkarthik
hkarthik11mo ago
seems like I might need to switch to poetry or read up on that issue looks like Pip had a breaking change
Brody
Brody11mo ago
but we still have one more thing to do run gunicorn and huey with parallel
hkarthik
hkarthik11mo ago
ah I didn't realize that was an option
Brody
Brody11mo ago
anything is possible if you know your way around nixpacks
hkarthik
hkarthik11mo ago
any docs I can look at?
Brody
Brody11mo ago
no need, i am docs, one sec nixpacks.toml
[phases.setup]
nixPkgs = ['...', 'parallel']
[phases.setup]
nixPkgs = ['...', 'parallel']
railway.json
{
"$schema": "https://schema.up.railway.app/railway.schema.json",
"build": {
"builder": "NIXPACKS"
},
"deploy": {
"startCommand": "parallel --ungroup --halt now,fail=1 ::: 'gunicorn wsgi:gunicorn_app' 'flask run-huey'"
}
}
{
"$schema": "https://schema.up.railway.app/railway.schema.json",
"build": {
"builder": "NIXPACKS"
},
"deploy": {
"startCommand": "parallel --ungroup --halt now,fail=1 ::: 'gunicorn wsgi:gunicorn_app' 'flask run-huey'"
}
}
you know what to do
hkarthik
hkarthik11mo ago
okay awesome thank you! really appreciate the help. Do you work for Railway?
Brody
Brody11mo ago
for a tiny explnation on parallel: run gunicorn and huey in parallel, this will fail fast if either service crashes allowing railway to restart the deployment i do not
hkarthik
hkarthik11mo ago
wow, well thank you so much either way anything I can do for you? venmo you a coffee or something?
Brody
Brody11mo ago
if you absolutely insist, i have a buymeacoffee in my bio, please dont feel obligated, i dont do this expecting anything in return
hkarthik
hkarthik11mo ago
oh I insist, you have some coffee coming your way. Thank you again!
Brody
Brody11mo ago
i set it to trains 🤣
Brody
Brody11mo ago
all running with parallel now?
hkarthik
hkarthik11mo ago
yes! it worked on the first try I'm gonna try to debug this pip issue later
Brody
Brody11mo ago
sounds good! might be good to have a read of this page as well https://nixpacks.com/docs/providers/python