R
Railwayโ€ข14mo ago
beuz

Keycloak unstable

Hello! So, I'm a bit of a newcomer to both keycloak and railway. I digged a little here in this group until I found a post by brody with some instructions on how to run keycloak on railway and eventually I managed to deploy my own instance and configure it. It works fine and I can login on my frontend app using next-auth. However, sometimes I just get a "application failed to respond" message but I can't find any errors if I look at the deploy logs at railway. I'm a bit lost and I don't know where I could start looking for errors. If it helps, here's the dockerfile I'm using: https://github.com/leonardochappuis/keycloak-docker/blob/master/Dockerfile
GitHub
keycloak-docker/Dockerfile at master ยท leonardochappuis/keycloak-do...
Contribute to leonardochappuis/keycloak-docker development by creating an account on GitHub.
No description
103 Replies
Percy
Percyโ€ข14mo ago
Project ID: N/A
beuz
beuzโ€ข14mo ago
a91e3f70-1457-4ae5-b646-23465ac28091
Brody
Brodyโ€ข14mo ago
i dont actually use keycloak myself, i only fixed some common issues with their dockerfile that prevented it from deploying, i think @ThallesComH does though, thalles, have any ideas?
ThallesComH
ThallesComHโ€ข14mo ago
i was having that issue and I just got back to hosting my own keycloak maybe it's that postgres timeout thing? i didn't dig deep enough
Brody
Brodyโ€ข14mo ago
looks like neither of us know, sorry ๐Ÿ˜ฆ
beuz
beuzโ€ข14mo ago
I can try to deploy again with a higher timeout but I'm not sure that's the issue. it usually works for a few requests and then I get that "application failed to response" screen for a few seconds I'd say it works like 6/10 times
ThallesComH
ThallesComHโ€ข14mo ago
yeah i was getting that too it doesn't show any error in logs i presume?
beuz
beuzโ€ข14mo ago
do you know if there's a way to output more uh, verbose logs?
ThallesComH
ThallesComHโ€ข14mo ago
for me it was the same thing
beuz
beuzโ€ข14mo ago
no
ThallesComH
ThallesComHโ€ข14mo ago
i don't remeber if i could find a variable to do that, but it's possible i think
beuz
beuzโ€ข14mo ago
oh, just found out ok, I'll try to deploy again with a higher timeout and with all logs btw I'm also from soccer country huehue obrigado pela ajuda! well, I'm not sure why but it appears you were right I increased the timeout by a high amount and I'm getting no more errors at least for now 10/10 requests going through I'm not sure if it will stay like this but if it does, I'll try to make a template
Brody
Brodyโ€ข14mo ago
yeah do template!!!
beuz
beuzโ€ข14mo ago
it'd be my first but I'm looking forward to it railway is the best
ThallesComH
ThallesComHโ€ข14mo ago
i've a dockerfile for keycloak, if you want i can send to you
beuz
beuzโ€ข14mo ago
thank you both for the help! I have managed to do a lot of stuff just by searching here and looking at your previous answers brody sure, that would help, thanks
ThallesComH
ThallesComHโ€ข14mo ago
FROM quay.io/keycloak/keycloak:22.0.1 as builder

ARG KC_DB_PASSWORD
ARG KC_DB_URL
ARG KC_DB_USERNAME

ENV KC_HEALTH_ENABLED=true
ENV KC_METRICS_ENABLED=true
ENV KC_DB=postgres
ENV KC_PROXY=edge
ENV KC_DB_PASSWORD=$KC_DB_PASSWORD
ENV KC_DB_URL=$KC_DB_URL
ENV KC_DB_USERNAME=$KC_DB_USERNAME
# Make sure to save the data before the container is stopped
ENV QUARKUS_TRANSACTION_MANAGER_ENABLE_RECOVERY=true

WORKDIR /opt/keycloak

RUN /opt/keycloak/bin/kc.sh build

FROM quay.io/keycloak/keycloak:22.0.1 as runner

ARG KC_DB_PASSWORD
ARG KC_DB_URL
ARG KC_DB_USERNAME

ENV KC_HEALTH_ENABLED=true
ENV KC_METRICS_ENABLED=true
ENV KC_DB=postgres
ENV KC_PROXY=edge
ENV KC_DB_PASSWORD=$KC_DB_PASSWORD
ENV KC_DB_URL=$KC_DB_URL
ENV KC_DB_USERNAME=$KC_DB_USERNAME
# Make sure to save the data before the container is stopped
ENV QUARKUS_TRANSACTION_MANAGER_ENABLE_RECOVERY=true

COPY --from=builder /opt/keycloak/ /opt/keycloak/

ENTRYPOINT ["/opt/keycloak/bin/kc.sh", "start", "--optimized"]
FROM quay.io/keycloak/keycloak:22.0.1 as builder

ARG KC_DB_PASSWORD
ARG KC_DB_URL
ARG KC_DB_USERNAME

ENV KC_HEALTH_ENABLED=true
ENV KC_METRICS_ENABLED=true
ENV KC_DB=postgres
ENV KC_PROXY=edge
ENV KC_DB_PASSWORD=$KC_DB_PASSWORD
ENV KC_DB_URL=$KC_DB_URL
ENV KC_DB_USERNAME=$KC_DB_USERNAME
# Make sure to save the data before the container is stopped
ENV QUARKUS_TRANSACTION_MANAGER_ENABLE_RECOVERY=true

WORKDIR /opt/keycloak

RUN /opt/keycloak/bin/kc.sh build

FROM quay.io/keycloak/keycloak:22.0.1 as runner

ARG KC_DB_PASSWORD
ARG KC_DB_URL
ARG KC_DB_USERNAME

ENV KC_HEALTH_ENABLED=true
ENV KC_METRICS_ENABLED=true
ENV KC_DB=postgres
ENV KC_PROXY=edge
ENV KC_DB_PASSWORD=$KC_DB_PASSWORD
ENV KC_DB_URL=$KC_DB_URL
ENV KC_DB_USERNAME=$KC_DB_USERNAME
# Make sure to save the data before the container is stopped
ENV QUARKUS_TRANSACTION_MANAGER_ENABLE_RECOVERY=true

COPY --from=builder /opt/keycloak/ /opt/keycloak/

ENTRYPOINT ["/opt/keycloak/bin/kc.sh", "start", "--optimized"]
ignore the custom theme thing qualquer coisa sรณ chamar
beuz
beuzโ€ข14mo ago
Ok, I believe it should be working. At least couldn't get it to fail so far ๐Ÿ˜‚ Far from perfect I guess but we now have a starting point https://railway.app/template/mSwigX?referralCode=AkM2z4
Railway
Deploy Keycloak on Railway
Keycloak template with keywind theme + apple and discord providers
Brody
Brodyโ€ข14mo ago
https://github.com/leonardochappuis/keycloak-docker/blob/master/Dockerfile#L27 while the variable will probably never be fully deprecated, it would be best to use RAILWAY_PUBLIC_DOMAIN instead of RAILWAY_STATIC_URL instead of 1StrongAdminPassword you can use ${{secret()}} to have railway generate a random 32 character string for every template deploy, you can also leave the admin variable blank so it forces the user to make their own
beuz
beuzโ€ข14mo ago
Ah, I didn't know that. I have updated both the dockerfile and the template. Hopefully it's better now thank you for the tips
Brody
Brodyโ€ข14mo ago
awsome, thank you for the template!!
beuz
beuzโ€ข14mo ago
no worries, thank you for your help! btw brody, last thing do you think I should change the postgres on my template to the new template one or is it ok to stick with the plugin one?
Brody
Brodyโ€ข14mo ago
great question, stick with the plugin version of postgres, as the template postgres uses tcp proxying and thats a beta feature as far as im aware, since ive seen non beta users have issues deploying templates that used beta features
beuz
beuzโ€ข14mo ago
aight, thank you again it did work on my end (non beta user here) but I guess it's better to be on the safe side ๐Ÿ˜…
Brody
Brodyโ€ข14mo ago
for sure!
beuz
beuzโ€ข14mo ago
unfortunately there's still something off
beuz
beuzโ€ข14mo ago
No description
beuz
beuzโ€ข14mo ago
really wanted to fix this
Brody
Brodyโ€ข14mo ago
kuma! and theres nothing bad in the logs when you get a 503?
beuz
beuzโ€ข14mo ago
no that's what's weird yeah, kuma is pretty cool I uploaded what's basically the same dockerfile to fly.io provisioned a high availability postgres, scaled the machine to 1gb ram
beuz
beuzโ€ข14mo ago
and there it's sitting like
No description
beuz
beuzโ€ข14mo ago
I really wish I could host it at railway tho
Brody
Brodyโ€ข13mo ago
there's gotta be something we are doing wrong, while railways postgres isn't highly available, they aren't high unavailable like your tests have shown
ThallesComH
ThallesComHโ€ข13mo ago
if you could diagnose this further would help a lot. can you try hosting the keycloak at railway and the postgres database at another provider? actually, nvm, i'll try it myself. so curious to see what's going on
Brody
Brodyโ€ข13mo ago
๐Ÿ‘€
ThallesComH
ThallesComHโ€ข13mo ago
i spinned up a Caddy webserver to proxy the requests to Keycloak through internal network and it's working great
No description
ThallesComH
ThallesComHโ€ข13mo ago
I've no idea tho what's happening, maybe it's related to ipv4 or ipv6, i've no idea @beuz if you want a workaround for now, spin up Caddy and use this Caddyfile
:{$PORT}

reverse_proxy http://keycloak-docker.railway.internal:8080
:{$PORT}

reverse_proxy http://keycloak-docker.railway.internal:8080
Brody
Brodyโ€ข13mo ago
hmmm what's the caddy for?
ThallesComH
ThallesComHโ€ข13mo ago
as i said, no idea, it's just working
Brody
Brodyโ€ข13mo ago
interesting, though I don't think I'm happy with that solution
Brody
Brodyโ€ข13mo ago
made some changes, will keep you both posted!
No description
Brody
Brodyโ€ข13mo ago
I got a 503 too, I am not built different
Brody
Brodyโ€ข13mo ago
bruh
No description
Brody
Brodyโ€ข13mo ago
414+ messages of keycloak down, followed by keycloak up
ThallesComH
ThallesComHโ€ข13mo ago
are you using caddy? kinda confused
Brody
Brodyโ€ข13mo ago
no ofc not
ThallesComH
ThallesComHโ€ข13mo ago
ah ok
ThallesComH
ThallesComHโ€ข13mo ago
and btw, it's still going strong
No description
ThallesComH
ThallesComHโ€ข13mo ago
ignore the first ones, configuration file skill issue
beuz
beuzโ€ข13mo ago
Hi guys, I'll try this as soon as possible Currently visiting my gf family Will be back in 2 days But thank you so much for the help!
Brody
Brodyโ€ข13mo ago
i seriously wouldnt recommend throwing caddy in front of it
beuz
beuzโ€ข13mo ago
I figured it could have something to do with that but no idea why tho The worst kind of bug. The one you have fixed but you have no idea why it works ๐Ÿ˜‚
Brody
Brodyโ€ข13mo ago
its likely just masking a problem, so it isnt a solution i can recommend
beuz
beuzโ€ข13mo ago
What I mean was that I was going to try connecting to an external postgres
Brody
Brodyโ€ข13mo ago
maybe keycloak is sending a malformed response, caddy can deal with it no problems and envoy (railways proxy) can't and ends up returning a 503 I'm pretty stuck on this as the reason, since if keycloak wasn't responding at all then it's not like caddy would return 200 while keycloak doesn't respond @ThallesComH what ya think?
ThallesComH
ThallesComHโ€ข13mo ago
we can probably test it https://hub.docker.com/r/envoyproxy/envoy if you've time to test it now, i can only do it at night
Brody
Brodyโ€ข13mo ago
hmmmm how easy is it to run envoy?
ThallesComH
ThallesComHโ€ข13mo ago
i've no idea tbh it's probably as difficult as traefik
Brody
Brodyโ€ข13mo ago
seems hard yeah traefik is hard well here's another reason for railway to expose envoys logs to the users (they already want to do that, but it's low priority so they haven't gotten around to it yet) Welp keycloak has not failed for a day or so now, without a separate railway service for caddy
Brody
Brodyโ€ข13mo ago
No description
beuz
beuzโ€ข13mo ago
how did you manage brody? to make it not fail for a day?
Brody
Brodyโ€ข13mo ago
with caddy
beuz
beuzโ€ข13mo ago
oh
Brody
Brodyโ€ข13mo ago
to be clear, I'm fine with using caddy, I just didn't like the idea of using it without knowing why
beuz
beuzโ€ข13mo ago
sure is it too difficult to import data from one keycloak instance to another or is it just a matter of importing the tables into a new postgres instance?
Brody
Brodyโ€ข13mo ago
I also wasn't fine with using caddy as a separate service because that's not fully backwards compatible with current deployments of keycloak
beuz
beuzโ€ข13mo ago
yeah, it would make things more difficult can you walk me through what you guys did? I'll update my template
Brody
Brodyโ€ข13mo ago
why are wanting to move keycloak data from one database to another? are you wanting to use the database services instead of the plugins?
beuz
beuzโ€ข13mo ago
no, I have my instance set up at fly.io still I have like $500 in credits
Brody
Brodyโ€ข13mo ago
haha did you apply for a job there?
beuz
beuzโ€ข13mo ago
yes
Brody
Brodyโ€ข13mo ago
nice free credit hack
beuz
beuzโ€ข13mo ago
I didn't get it but still would want to make use of those credits xD hahahah yeah
Brody
Brodyโ€ข13mo ago
well you could use pg_dump and pg_restore
beuz
beuzโ€ข13mo ago
I'd want to move the data to railway once I'm out of credits though yeah, that's what I thought. I believe it'd work, they're basically the same images, configuration and everything
Brody
Brodyโ€ข13mo ago
yes I'll provide you a doc with all the changes you need to make, and I'll do a pr on your template repo that adds my modified dockerfile and caddyfile you could probably even use a desktop tool like dbgate that has a GUI and can clone over a database for you
beuz
beuzโ€ข13mo ago
right, sounds easy enough and thank you for the pr btw
Brody
Brodyโ€ข13mo ago
and I'll get all the information you need in order to update the template and the pr later today
beuz
beuzโ€ข13mo ago
That was my first time using keycloak so I was never sure if I was doing something wrong I mostly used external auth like firebase or auth0 exclusively
Brody
Brodyโ€ข13mo ago
I'm pretty confident in this being an issue with how keycloaks http server is interacting with railways proxy
beuz
beuzโ€ข13mo ago
yeah, that makes sense as to why caddy would fix it
Brody
Brodyโ€ข13mo ago
ideally keycloaks http server would not be sending malformed responses, and Ideally railways envoy proxy should be a little more forgiving, so both sides are at fault caddy is built in go, and go apps just simply work without any fuss
beuz
beuzโ€ข13mo ago
true this I can neither confirm nor deny but I'll trust you
Brody
Brodyโ€ข13mo ago
and what do you think about not exposing the postgres service publicly by default? because in my working implementation I use the private network to communicate to the database without having the database exposed publicly (this isn't helping keycloak work better, it's just for an extra piece of mind)
beuz
beuzโ€ข13mo ago
sure, that would work it's not used in anything else
Brody
Brodyโ€ข13mo ago
the only thing I can think of for having it exposed publicly would be importing data into it
beuz
beuzโ€ข13mo ago
that requires it to be public yeah, well, that's an edge case, I think
Brody
Brodyโ€ข13mo ago
yeah that's why I like to add a note to the overview that states this information and how to turn the tcp proxy on and back off
beuz
beuzโ€ข13mo ago
I'll do just that
Brody
Brodyโ€ข13mo ago
I'll write that for you too
beuz
beuzโ€ข13mo ago
thank you sensei
Brody
Brodyโ€ข13mo ago
https://github.com/leonardochappuis/keycloak-docker/pull/1 there are example env files, all the variables would need to be updated on the templates service, if i dont have the variable in my example env files, then you dont need it on the service, for example you have a PORT variable, and i dont, so you can remove it from the template. please change the service name from "Keycloak-Docker" to just "Keycloak" your template uses the old postgres plugin, remove that from your template and add another service named "Postgres" keep the network unexposed, image source is ghcr.io/railwayapp-templates/postgres-ssl:latest and volume mount point is /var/lib/postgresql/data then just use the environment variables from the example env file in the keycloak service set the healthcheck path as /health/ready add this to your template overview
**Things of Note:**

- Communication to Postgres is done exclusively over the private network and the database is not exposed externally in any way by default, if you want to enable access from outside of the private network you can go to the databases settings to enable TCP proxying and enter the internal port (5432). the TCP proxy can be removed at any point to close off external access.
**Things of Note:**

- Communication to Postgres is done exclusively over the private network and the database is not exposed externally in any way by default, if you want to enable access from outside of the private network you can go to the databases settings to enable TCP proxying and enter the internal port (5432). the TCP proxy can be removed at any point to close off external access.
i probably forgot something, so let me know if something doesnt make sense
beuz
beuzโ€ข13mo ago
Hey brody! thank you for this I'll make the changes now Ok, everything should be live oh no, I'm missing the healtcheck path but other than that it should be ok aight, all set yes
Brody
Brodyโ€ข13mo ago
leave KEYCLOAK_ADMIN blank you forgot to fill out POSTGRES_HOST and POSTGRES_PORT with the reference variables I gave I do see what you where getting at from your description of those variables, but that's not quite how it works, you can fill out those reference variables and set the service to unexposed in the template editor mode, so that when/if the user wants to enable tcp proxying they can do so via the postgres services settings menu you should join beta, type /beta @beuz ๐Ÿ™‚
beuz
beuzโ€ข13mo ago
done and done and done
Brody
Brodyโ€ข13mo ago
everything looks good from just the template deploy page, have you done another test deploy of this template?
beuz
beuzโ€ข13mo ago
just did
No description
No description
Brody
Brodyโ€ข13mo ago
everything look good?
beuz
beuzโ€ข13mo ago
yep, so far so good
Brody
Brodyโ€ข13mo ago
awesome
beuz
beuzโ€ข13mo ago
yeah, I'm glad
Brody
Brodyโ€ข13mo ago
thank thalles, he came up with the idea I just built upon it
beuz
beuzโ€ข13mo ago
for sure thank you @ThallesComH, now everyone can use keycloak hassle free yes
Want results from more Discord servers?
Add your server