Wasp•2y ago

Problems with server on fly.io - unexpectedly without any updates or redeployments

Hello, I'v had some problems with my rooms booking wasp app, running on fly.io. It has been running perfectly for four months (bookd.xyz / foretagscenter-client.fly.dev), but yesterday I noticed that it had stopped working, and I am puzzled by the cause of it.

Problem:

It stopped working out of thin air. No redeployments or anything.
The client does not get any response from the server.
Server logs look like attached - (GET 401 /auth/me)
No error logs from postgres database - no strange behaviour when reviewing the server metrics (I have very few users)

Tried solutions:

Redeploying the app with wasp deploy fly deploy
Restarting the server with wasp deploy fly cmd --context server scale count 0 - wasp deploy fly cmd --context server scale count 1
Restarting the postgres db with flyctl postgres restart -a foretagscenter-db

Troubleshooting steps

Run the app locally works fine (wasp start)
Run the app but connected to fly.io db through proxy works fine

Other than that I am not certain on what to do next. Any possible help or ideas would be highly appreciated!

Best regards
Frej

martinsos•1/30/24, 3:25 PM

@frej1046 , one thing to to explore -> why does https://bookd.xyz/ not work at all? So if I go to https://foretagscenter-client.fly.dev/, I get an app that is erroring out because server is rejecting it due to CORS errors, which actually make perfect sense since your server is probably configured to accept requests only from https://bookd.xyz/ (am I making sense here @Wasp Team ?).
But, your https://bookd.xyz/ doesn't resolve to a page at all, I just get This site can't be reachedThis site can't be reached by Chrome, with ERR_CONNECTION_CLOSEDERR_CONNECTION_CLOSED error. Any idea why is that happening? Sounds like an issue with your domain possibly, since Fly's client URL works?

miho•1/30/24, 3:29 PM

Have you setup a CNAME record? https://fly.io/docs/networking/custom-domain/

Fly

Use a custom domain

Documentation and guides from the team at Fly.io.

miho•1/30/24, 3:30 PM

Checking your domains CNAME record here doesn't show it https://dnschecker.org/#CNAME/Bookd.xyz

DNS Checker

DNS Checker - DNS Check Propagation Tool

Check DNS Propagation worldwide. DNS Checker provides name server propagation check instantly. Changed nameservers so do a DNS lookup and check if DNS and nameservers have propagated.

Vinny•1/30/24, 4:19 PM

The other solution I could propose is that you upgrade your app to the newest Wasp version. I had a similar issue with Railway. They changed something on their end and a recent update we pushed fixed it (cc: @miho ). I just had to:

run curl -sSL https://get.wasp-lang.dev/installer.sh | shcurl -sSL https://get.wasp-lang.dev/installer.sh | sh

change the wasp version on my app in

main.wasp

main.wasp

wasp: {
 version: "^0.11.8"
},

wasp: {
 version: "^0.11.8"
},

and then make sure everything worked locally
and then redeployed with wasp deploy fly deploywasp deploy fly deploy

VVinny The other solution I could propose is that you upgrade your app to the newest Wa...

martinsos•1/30/24, 7:23 PM

This is a good idea also! Although I would keep it as the last option, if we run out of ideas, so we don't introduce another variable into the whole debugging process.

Vinny•1/30/24, 7:51 PM

true

Mmiho Have you setup a CNAME record? https://fly.io/docs/networking/custom-domain/

frej1046OP•1/31/24, 8:09 AM

Yes, this I could try of course. When setting it up I followed the guide at https://wasp-lang.dev/docs/advanced/deployment/cli#using-a-custom-domain-for-your-app, without adding any CNAME record. It worked until now, but things might change on their side.

I will try to follow the fly.io documentation for this one.

Deploying with the Wasp CLI | Wasp

Wasp CLI can deploy your full-stack application with only a single command.

frej1046OP•1/31/24, 8:11 AM

https://community.fly.io/t/cert-not-being-issued-after-12h/9795/24 This might have something to do with it. ~~I did not have a AAAA record set up (only A record as per the documentation)~~, EDIT: Sorry, I did have the AAAA record, as per the documentation. and fly.io have apparently made some updates here. I will continue with trying to recreate the certificate and redo the DNS settings using A and AAAA posts. (My DNS provider does not allow me to put CNAME under @

Fly.io

cert not being issued after 12h

A fix has been rolled out. Sorry about all of that, there are more rough edges with shared IPs than we initially thought.

frej1046OP•1/31/24, 9:18 AM

issuing a new certificate did the trick!!

Removing the old certificate fly certs remove bookd.xyz -a foretagscenter-clientfly certs remove bookd.xyz -a foretagscenter-client
Creating a new certificate fly certs create bookd.xyz -a foretagscenter-clientfly certs create bookd.xyz -a foretagscenter-client
Adding the proper DNS records accoring to logs (A and AAAA)
Watch progress fly certs show bookd.xyz -a foretagscenter-clientfly certs show bookd.xyz -a foretagscenter-client
Repeat for www (but with CNAME post on www subdomain)

Thank you all so much for the help, I highly appreciate it!

MEE6•1/31/24, 9:18 AM

Wohooo @frej1046, you just became a Waspeteer level 1!

martinsos•1/31/24, 9:46 AM

Awesome @frej1046 , I am glad we figured it out!

Do you think there is something that needs updating in our docs?

frej1046OP•1/31/24, 9:54 AM

No, not at all. The docs are fine. This seems to have been a fly.io thing, where they struggled with the certificates for shared IPs.

Ffrej1046 No, not at all. The docs are fine. This seems to have been a fly.io thing, where...

martinsos•1/31/24, 10:09 AM

Got it, thanks :)!

Problems with server on fly.io - unexpectedly without any updates or redeployments

Similar Threads

Similar Threads

Similar Threads