Postgres Migration taking a long time, can I still use the database it's migrating from?
Hey, db has been migrating for over an hour and some production services are down. Could use some help here
48 Replies
Project ID:
db735447-598c-455d-b1be-6718e98e1affdb735447-598c-455d-b1be-6718e98e1aff
@fp - database migration stuck
Hmm, I think the migration is complete but the workflow is failing to detect the same.
Did you manually make any changes to the migration service or the new DB?
production is totally busted right now
i did not make any manual changes
Okay, so I think the migration went fine and our system failed to detect the same. I've manually canceled the migration.
but i did deploy a service that runs a prisma migration
while it was migrating
yeah my bad
Could you verify the data was moved properly and then just port over the variables?
I think you should be good.
it looks like the data is good yes
Basically, any service that uses the legacy db, point it to the new db.
going to do that right now, will get back to you
yup
@Brody feel free to loud tag me as I am stepping away from the desk for dinner.
updated everything, still cant connect
but I can connect locally with the connectionstring
so something must be wrong on my setup
what errors are you getting
cant reach database server at 'viaduct.proxy.rlwy.net':'23653'
can you add a 3 second sleep to the beginning of your start commands?
yeah that didn't change it, it appears to be a problem with the connection string
probably something really stupid
can you show me how youve added the sleep?
should i be using the private url for postgres?
Where should the sleep be?
i just put a 3 second sleep befre starting the app. Its an express app
if you dont want to be charged for egrees, yes
can you show me how youve added the sleep?
like i said, its a 3 second await
in the main function, before starting the application
in the start command please
on railway?
whatever is most applicable, that or in the start script of your package.json
"start": "sleep 3 & prisma migrate deploy && node build/server.js",
this doesnt actually sleep for 3 seconds
better way to do this?
why wouldnt that sleep for 3 seconds?
locally it just rips right through it, i dont know
ill try and deploy
can I have two different postgres instances in the same project?
or do the postgres variable references overlap?
a bit confused aboutthat
absolutely, the references can't overlap because you can't have two services with the same names
yup i can see that now
sleep didnt fix it
im certain its an issue with the connection string
I don't know how, but it is
can you connect to the database locally
yes
i know the problem, but i dont know why its happening
are you sure railway is using the start script with the sleep?
yes, and now im getting a prisma migration error
so it's actually connecting to db
hold on
doing a migration rollback...
when does the service run the migration?
and are you trying to use the private url now?
Im using the private url on prod
but when i run the migration on the prod db locally, im using the public
start script is sleep for 3, migrate, and run
can you maybe show me why you are sure railway is running a start command with the sleep in it
I fixed it
Whew. Thanks for the help
it was 2 issues. Sleep helped it, plus the migration issue
i can delete these old legacy dbs right?
as long as you have confirmed your data is where it should be
deleting them, prod is stable and online. Thank youa gain
glad to hear it!
why does this fix this?
theres a private network init time
still getting it but only on one of my services
should i try upping the sleep time?
sure
sleep 8 & yarn start
not doing anything... ill try with a semi colon?
&&
whoops