Deployment failing since Railway incident report

I pushed a very small update to my api project, at the same time as a reported Railway incident. Since then, all my builds are failing, and I can't rollback to a previous build. I wouldn't think the two incidents would be related, but it's a bit strange in that the change was very small so it doesn't seem like a coincidence?

Project: ac828fce-6f54-43e9-825c-ac4871d226ba

Have you tried rolling back your code and pushing a new commit? If there's an issue on Railway's side, rolling back to a previous build may not work correctly.

im trying now. it very well could be an issue on my side. it's just weird that this happened at the same time as the incident. so i wanted to report it to see if others were having an issue.

so i rolled back successfully. i re-push the new code but that's not working. i'll try to revert my PRs

that actually won't work. so i have a stable release from before I saw the Railway Intermittent Issue notice. I pushed a small release after that and the deployment failed. I since rolled back my PR and pushed to Railway but that also failed

so to summarize:
- i had stable code in Railway
- i tried to push a release, which failed
- i then saw the Railway outage notification (after I tried to push the release)
- i tried to revert the code to the previously-stable version. this also failed.
- so now, i have a version of code running on hostway that is not the same as my current active code branch.

Interesting issue. What does the error say?

the healthcheck fails. and the deploy logs have hundreds of errors

what are those errors?

very low-level. e.g. pydantic.

le "pydantic/fields.py", line 506, in pydantic.fields.ModelField.infer
File "pydantic/fields.py", line 436, in pydantic.fields.ModelField.__init__
File "pydantic/fields.py", line 552, in pydantic.fields.ModelField.prepare
File "pydantic/fields.py", line 663, in pydantic.fields.ModelField._type_analysis
File "pydantic/fields.py", line 808, in pydantic.fields.ModelField._create_sub_type
File "pydantic/fields.py", line 436, in pydantic.fields.ModelField.__init__
File "pydantic/fields.py", line 552, in pydantic.fields.ModelField.prepare
File "pydantic/fields.py", line 668, in pydantic.fields.ModelField._type_analysis
File "/usr/local/lib/python3.9/typing.py", line 852, in __subclasscheck__
return issubclass(cls, self.__origin__)
TypeError: issubclass() arg 1 must be a class

Looks like a code issue

but i made a really small code change (in a test case). and then reverted the PR

Yeah sorry this isn't an issue with Railway. It's 100% an issue with your code

Your app is building, but it's failing during deployment. That's indicative of a code issue

the app is failing during the build process

full build logs please?

sure. how would you like me to send? do you just want to look at the service via the service id?

only team members can look at the service, i have access to nothing

so in a .txt file please

attached build and deploy

full build logs please

wait

why is your image 4.9 gb

torch, etc

can you send your requirements.txt

oh you should really be pinning your packages to specific versions

one of those packages must have made a breaking change

ugh

i can try to rebuild in a fresh env locally

also im sorry to tell you this but using duckduckgo_search breaks tos

as it scrapes duckduckgo

i can disable that for now. not really using

much appreciated, thank you for understanding

so you have seen this sorta issue before i assume?

yes

though i do not know what specific package was updated that results in this issue

i have a guess

let me know if you figure it out, so i can pass on this info to anyone who may have this issue in the future

long shot, but?
https://stackoverflow.com/a/76313951

ha. yeah, this was my guess 🙂 thanks for sending this. im gunna pin that version

keep me updated 🙂

ill let you kow if that works

for sure!

i assume this works locally, just pin the packages in the requirements.txt file to the versions you have installed on your computer? would that be possible?

worked!! 🙂

thanks a lot for the help!

latest langchain is 0.0.178?

had planned on pinning everything never got around to it. will make it a priority now 🙂

yeah, they update their package prob every day. i had 0.0.173 locally. 0.0.178 broke the build. prob 177 is ok as the build was working yesterday

oh 9.9.178 is typo lol

so your app worked as soon as you pinned langchain to 0.0.173, did you need to pin anything else?

no- just pinned that. wanted to try that first, then start pinning other stuff if it didnt fix it. i figured it was langchain as much of my code depends on their types, and the errors were pydantec related. and they update about every day. i'm in their discord- i can ping them and let them know

perfect, thank you, this info will be used to help others who may run into this issue

do you think you could fix the 9.9. typo so i can mark your message as the correct answer?

sure- whre is that typo?

^

done

thanks!

thank you!