Degraded deploy speeds?

Has anyone noticed deploys are going out pretty slow? I'm deploying vanilla Go servers, no docker or special NIX dependencies required, but I'm seeing deploy times up to several minutes. Sometimes they go by quickly, <1 minute, and sometimes it takes several minutes per deploy. If an admin sees this, here is a relevant build: https://railway.app/project/b508b3ae-a488-4863-a24f-3990f5b88fb3/service/78a4c936-e42e-48be-9966-f5ba72a9b7db?id=54363a84-2429-4ab7-843d-150971d5699c It's the container step that's taking much longer that I'm used to.
Railway
Railway
Railway is an infrastructure platform where you can provision infrastructure, develop with that infrastructure locally, and then deploy to the cloud.
No description
28 Replies
Percy
Percy3d ago
Project ID: b508b3ae-a488-4863-a24f-3990f5b88fb3,78a4c936-e42e-48be-9966-f5ba72a9b7db,54363a84-2429-4ab7-843d-150971d5699c
zgmg92
zgmg923d ago
Logs if it helps
Brody
Brody3d ago
yeah deploy times are not the best right now, the team is working on speeding them up. cc @Angelo
zgmg92
zgmg923d ago
Tried redeploying and immediately saw this
No description
zgmg92
zgmg923d ago
zaydek@Zaydeks-MacBook-Pro orderediting-go % railway up --service=headers-experimental --detach
Indexed
Compressed [====================] 100%
Failed Failed to upload code with status code 500 Internal Server Error
zaydek@Zaydeks-MacBook-Pro orderediting-go % railway up --service=headers-experimental --detach
Indexed
Compressed [====================] 100%
Failed Failed to upload code with status code 500 Internal Server Error
zaydek@Zaydeks-MacBook-Pro orderediting-go % railway up --service=headers-experimental --detach
Indexed
Compressed [====================] 100%
Failed Failed to upload code with status code 500 Internal Server Error
zaydek@Zaydeks-MacBook-Pro orderediting-go % railway up --service=headers-experimental --detach
Indexed
Compressed [====================] 100%
Failed Failed to upload code with status code 500 Internal Server Error
zgmg92
zgmg923d ago
The original deploy took about 7 minutes in deploy time alone
No description
zgmg92
zgmg923d ago
Thanks @Brody If I use a Dockerfile will it bypass the deploy container step or not necessarily? I'm seeing 1-10 minute deploys
Brody
Brody3d ago
container still has to be deployed regardless dockerfile, nixpacks, source image, etc
zgmg92
zgmg923d ago
Kk, thanks
angelo
angelo3d ago
Yea, the publish step is the same regardless
Flint
Flint3d ago
Hope to see this improve soon. Love railway, but these recent deployment times are really killin it for me.
rojwilco
rojwilco3d ago
experiencing the same issue here stuck on step Deploy > Create Container for 15 minutes, and then it timed out with activity StartToClose timeout
Brody
Brody3d ago
the team is actively working on improving this
rojwilco
rojwilco3d ago
can we get an announcement or something in the incidents channel when the fix is deployed?
Brody
Brody3d ago
its possible that the team will post some kind of post mortem
Garcia
Garcia3d ago
????? my application only dont work
Brody
Brody3d ago
please open your own thread as this seems unrelated to the topic of this thread.
Brody
Brody3d ago
thats a private thread, this topic does not need to be a private thread
angelo
angelo3d ago
Going to investigate- in this case, I don't mind. Point being is that the team, and esp. myself know how frustrating this whole experience is right now.
Garcia
Garcia3d ago
as we are pro, and because of what it says it was supposed to be faster, I opened it this way, as we have been stopped several teams because of this problem
Brody
Brody3d ago
i understand, but please know that it says that message no matter the option selected
Garcia
Garcia3d ago
ok
angelo
angelo3d ago
Okay @here, raised a post-facto incident, https://discord.com/channels/713503345364697088/846875565357006878/1261026504574369894 and we expect to have a fix to this in 4 hours. Early reports are looking like a host config issue, we plan to share more on what happened.
Duchess
Duchess2d ago
New reply sent from Help Station thread:
Angelo, if it's not too much trouble, could you re-share that incident on this forum or in a spot with more visibility?
You're seeing this because this thread has been automatically linked to the Help Station thread.
Duchess
Duchess2d ago
New reply sent from Help Station thread:
You can check the incidents here -https://status.railway.app/
You're seeing this because this thread has been automatically linked to the Help Station thread.
Railway - Status
Railway Status
Duchess
Duchess2d ago
New reply sent from Help Station thread:
@Brody / @angelo As of 9:18 PDT, the Railway status incidents page shows All Systems Operational. However, I'm still observing degraded deploy speeds. I've seen this across a few different apps including MySQL. Here are a couple of screenshots of a stuck MySQL deployment. Are you observing these issues on your end as well?
You're seeing this because this thread has been automatically linked to the Help Station thread. New reply sent from Help Station thread:
Both those screenshots show a single active deployment, in this case the deployment event stage listing as not started is likely just a bug.
You're seeing this because this thread has been automatically linked to the Help Station thread. New reply sent from Help Station thread:
@Brody I agree it's a bug. I want to add that another observation: when I push a new deployment, I need to manually remove the previous one. That suggests there's still an issue. Is the team planning to investigate this issue?
You're seeing this because this thread has been automatically linked to the Help Station thread. New reply sent from Help Station thread:
We are working on that one as we speak, does that deployment have a volume per chance?
You're seeing this because this thread has been automatically linked to the Help Station thread.
Duchess
Duchess2d ago
New reply sent from Help Station thread:
Thanks.Yes, it does. I had a hunch it was related to volumes and wrote up what I'm seeing here: https://help.railway.app/questions/issues-with-volumes-68850731
You're seeing this because this thread has been automatically linked to the Help Station thread.
Railway Help Station
Issues with volumes?
I'm building a new template that uses Wordpress and Volumes and I'm seeing the following issues:I am unable to delete an app that had an attached volume. The app is missing from the canvas, but still shows up as option.Attached volumes have incorrect permissions.Apps with Volumes appear to be causing deploys to be stuck in "Deployment in progres...