Deploy stuck
Got a Meilisearch deploy stuck, redeploy doesn't work
Solution:Jump to solution
Here is an update:
- we identified a bug which was causing region changes to be applied for all environments of a project instead of the currently selected one
- the region change for the non-current environment was just a value change in our db, not a real region change
- we have fixed that bug
- we have restored thew correct values in our db...
22 Replies
Project ID:
N/Ahappened after this deployment error
deployment context hash: 9ba5aef890ce3ab518ccbcfeccf21794
project id: 068c8b3c-c76e-4b92-81b1-2900a9605c90
service id: 7f28bdc4-c7e3-432f-90a0-420e3f2536dd
Thread has been flagged to Railway team by @Brody.
really need this fixed in the morning π π
right now this application is down because i was dumb enough to remove the active deployment
@thomas - builder issue, docker daemon failing to respond, would effect multiple users
moving to us-west solves it, i'll leave the service in there until railway team has a solution.
you had it on us-east before?
yes, on us-east it breaks
Did you go through the volume migration workflow?
Assuming this has a volume
i did a volume migration a while ago, but doesn't seems to be cause of this
actually, i only redeployed it because i enabled app sleeping in the dev environment and it applied to production
then i disabled app sleeping and redeployed just to make sure that it wasn't enabled in production
after that i got this
maybe the error isn't happening anymore? i'll try to move it to us-east again
got another symptom of the problem, i asked railway to move the volume to
us-east.
the development environment already migrated but the production didn't start anything yet
Aha! That's probably it!
It might have only migrated for one environment but marked it completed!
PLAT-786 - Voluem migration may not take into account multiple environments
User in attached thread has an issue where their volume was migration in one environment, but NOT another
Status
Triage
Platform
CC @borck for when they're up (eng who built it)
Wondering if the above sounds correct
ok this might sound weird but i don't remember if i migrated the meilisearch application (it was at 3am), but im like 70% that no.
that error just appearing out of nowhere when i tried to redeploy the app (no region migration was done).
but as a said, it might lead to the actual error
gm! I'm having a look at this now
I'm not sure I understand the context here, so let me first clarify some things and ask some questions:
- Volumes and services in different environments are independant. If you change the region of a service in one environment, it does not affect the region of the same service in other environments. The same is true for volumes.
- That means if you changed the region of a service with a volume attached in one environment, it has no effect on the same service/attached volume in other environments
- In other words, the region setting of a service is tied to a given environment. You can run a service in us-west in dev and in us-east in another environment
With that, let's go back to the issue you are facing.
I do see the following problem indeed:
- Your Meilisearch service in the
production environment is set to the us-east region but its attached volume somehow is still in us-west.
- The same service in the development environment is in the us-east region and the volume is correctly in us-east too
I'll investigate how the issue happened in your production environment.
Did you change the region in both production and development at the same time? I'm trying to reproduce the issue with one migration not going through.
I think I identified the problem to be this: when manually triggering simultaneous migrations of a volume in several environments, only one of them goes through properly.
Just FYI, The current situation for your case is that your Meilisearch in production is actually deployed in us-west despite the UI wrongly showing it in us-east. The volume is also in us-west.
This is likely a result of the failed migration of the volume.
I'll debug this and come back with a resolution once I have it.@ThallesComH
this is hard to explain, but that error did not occur because of a volume migration i think
I have found what is most likely the issue and am working on a fix.
Will update you when it is ready.
Solution
Here is an update:
- we identified a bug which was causing region changes to be applied for all environments of a project instead of the currently selected one
- the region change for the non-current environment was just a value change in our db, not a real region change
- we have fixed that bug
- we have restored thew correct values in our db
Concretely, your meilisearch service in
production was showing us-east as the region but was effectively deployed in us-west. This is now correctly reflected.
Apologies for the confusion π