Deployment Crashed emails
I'm pretty frustrated about a project of mine that keeps crashing. its an open source project who recommends railway, tbh im going to probably migrate off of it because its insane how often it crashes.
Is there a setting where instead of emailing me with a big Restart Deployment button, railway could just, ya know, restart it??
otherwise, can i pipe these alerts somewhere else? like telegram?
36 Replies
Project ID:
08eeb7e1-edff-4abc-beb9-a2a42017cf9008eeb7e1-edff-4abc-beb9-a2a42017cf90
by default railway will restart the deployment up to 10 times, after that no more restart attempts will be performed https://docs.railway.app/guides/healthchecks-and-restarts#restart-policy
what software, and what errors does it crash with?
project is https://ponder.sh
Ponder – A backend framework for crypto apps
Ponder is an open-source framework for crypto apps focused on developer experience and performance.
but ok we will just migrate off of them to a different provider, am sick of this
im sorry your experience with Railway hasn’t been as pleasant as you would like, but i would be happy to try to debug any errors you are getting
yeah i mean tbh i dont know and dont want to know or be responsible for this. id rather use a fully managed instance from a saas provider, not have to manage a deployment myself. its a very small project that needs to index some blockchain data in order to perform a daily cron task
totally understand, self managed is not always what everyone wants, you should do whats best for you
yeah was a bad decision in the first place, which is on me not railway!! thanks though, we tried!
sad to see you go, hopefully railway can be your host for a different project in the future
well im definitely done with heroku! so yeah, potentially. i might suggest reaching out to the ponder team about railway templates? feels like some other devs i know are in a similar spot where they are pretty frustrated and migrating off, so it imght be beneficial to railway to work with ponder? idk
i'd still be very interested to know what errors you are getting? of course only if you have the time?
i truly have no idea
happy to have you look at our logs but its just like, someone else wrote the software, i followed some isntructions to deploy on railway, and every few days i get an email that the service crashed
is there anything in the deployment logs?
could you also link the instructions?
admittedly we are a few versions behind, but its just a thing i dont want to think about and to be able to rely on it being up
railway deploy instructions - https://ponder.sh/docs/guides/production
Deploy to production – Ponder
A guide for deploying Ponder apps to production
the logs are pretty polluted with updates about indexing new data. and im not super familiar with railway observability
would it be safe for someone who doesnt work for railway to look at your logs? (me)
like i have no idea what to make of this
yeah absolutely, have at it!
that screenshot probably provides the error id need to see anyway, so i think im good with just that
is there a sample ponder app in a repo i can take a look at?
yeah they have some examples in their source repo - https://github.com/ponder-sh/ponder/tree/main/examples
GitHub
ponder/examples at main · ponder-sh/ponder
A backend framework for crypto apps. Contribute to ponder-sh/ponder development by creating an account on GitHub.
but again we are def a couple versions behind, which is on me, but they changed a lot of their config between versions so just havent gotten upgraded yet
what example would you say is closest to the app you are running?
tbh the thing that would help the most right now would be if we could pipe a message into telegram when things crash
unfortunaly railway doesn’t have a native way to do that kind of thing right now
yeah and i dont have the bandwidth to build something custom. its a tiny project that should just be on autopilot. but ppl depend on it. and when we miss it crashing, we screw over ppl who depend on it, and then making that right gets expensive for us
i understand, thats fair
but i have seen that error before, its when the database pool keeps disconnected connections in the pool and then tries to use the disconnected connection thats still in the pool, it doesnt look like ponder provides a way for the user to change this in code, so it would need to be something ponder changes in their codebase
would you be able to give me the full logs around the error? i dont think i can reliably copy out of the ui
i dont work for railway so i have no way of getting your logs unless you sent me them yourself, there is this tool to help you do that https://bookmarklets.up.railway.app/template-info/
oh
ok interesting
i misread the "doesnt work at railway" part
though a change you could do would be to use ponder with an sqlite database and a volume, but this comes with additional caveats, it would solve the postgres pool errors though
yeah i think i found that with sqlite whenever we redeployed it would rebuild the whole index, which as time goes on takes longer and longet
ideally we wont need to redeploy, but the postrgres instance helps that
yeah thats why you'd want to use a volume and store the sqlite database there so that the sqlite database persists between redeployments
theres absolutely no pressure to stay on railway, like i said you do whats best for you, but id be happy to help you get the volume setup
im pretty busy on other work, my collaborator is working with ponder dev to get us upgraded and see if that would help
so we'll see how that works out first i suppoese
no worries, let me know if you'd like to try the sqlite + volume approach, ill be around!