W
Windmill13mo ago
rubenf

CookieMonster 9682 it s not the wrong

@cookiemonster0189 it's not the wrong place. What is your issue ?
35 Replies
CookieMonster
CookieMonster13mo ago
I deployed Windmill using the provided helm chart to GKE but the pods are stuck in a crashloop with no additional logging on the containers. I assume it is because I cannot connect to SQL Cloud
rubenf
rubenf13mo ago
If it's because of that it should actually be mentionned in the logs
CookieMonster
CookieMonster13mo ago
using the following databaseUrl value postgres://postgres:testing1234@xxxxxx-test:us-central1:windmill-database-instance-fbe9ac64
rubenf
rubenf13mo ago
are you able to connect to the database using psql and that database url from your pod/node ?
CookieMonster
CookieMonster13mo ago
I have not gone that far yet, only because the cluster is completely private and I would need to punch through to the control plane for debugging. The SQL instance is alive and well
CookieMonster
CookieMonster13mo ago
No description
CookieMonster
CookieMonster13mo ago
this is a just a test deploy, this is not hardend yet
rubenf
rubenf13mo ago
first i 'd like to confirm it's the database url are there 0 logs ?
CookieMonster
CookieMonster13mo ago
pods deploying right now. Ill get back to you on the logs in the jsut a sec. Also Ill start down psql right now 🙂 please hold elevator music
CookieMonster
CookieMonster13mo ago
@rubenf I was able to connect to the SQL database successfully
CookieMonster
CookieMonster13mo ago
@rubenf Still struggling. I think the issue is still GKE / SQL Cloud related. GKE is kinda weird in the sense, that in-order for a private autopilot GKE cluster to connect and talk to Google Cloud Services in another VPC; SQL Cloud it recommends running a sidecar 😬 SQL Auth Proxy container inside the same pod as the application so that it can connect over localhost. This allows a kubernetes service account to talk to Google's Private Cloud Services (SQL Cloud) in a different VPC via network peering and using a federated Workload Identity. I noticed that I can provide a diffrent Kubernetes Service Account to the provideded Helm Chat; which is good. This will allow me to create a service account with the proper roles, and bindings. My only question at this point is it possible to deploy another container to the same pod using Windmill's Helm Chart? Otherwise I think I might just have to reverse engineer the chart and do my own deployment. I am still new to GKE 😭. Also... any GCP experts here? What am I missing? Any help would me much appreciated. https://cloud.google.com/sql/docs/postgres/connect-kubernetes-engine#proxy https://cloud.google.com/sql/docs/mysql/sql-proxy https://github.com/GoogleCloudPlatform/cloud-sql-proxy
GitHub
GitHub - GoogleCloudPlatform/cloud-sql-proxy: A utility for connect...
A utility for connecting securely to your Cloud SQL instances - GitHub - GoogleCloudPlatform/cloud-sql-proxy: A utility for connecting securely to your Cloud SQL instances
rubenf
rubenf13mo ago
Yeah for the sidecar you will probably have to fork the chart Or write a deamon set Also the helm charts support running postgresql inside kubernetes (just need to be enabled and it will spawn the bitnami chart). Maybe that's what you need?
CookieMonster
CookieMonster13mo ago
We are using Pulumi as IaC. I think there might be a really trivial way to transform the existing chart into a new deployment. If your interested I can keep ya posted. https://www.pulumi.com/docs/using-pulumi/adopting-pulumi/migrating-to-pulumi/from-kubernetes/#configuration-transformations
pulumi
Kubernetes YAML & Helm Charts
Migrate your existing Kubernetes YAML or Helm Charts and/or coexist with existing templates.
rubenf
rubenf13mo ago
That would be great, thanks @cookiemonster0189 any progress ? Can I help on anything ?
CookieMonster
CookieMonster13mo ago
Awe thanks for following up @rubenf! I am actually working on it again this morning. I'll make sure to follow-up. Got distracted working on deploys yesterday 😬
CookieMonster
CookieMonster13mo ago
@rubenf no luck so far... not seeing any logs from the container as to what is causing the crashloop.
conditions:
- lastProbeTime: null
lastTransitionTime: "2023-05-25T01:56:57Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2023-05-25T01:56:57Z"
message: 'containers with unready status: [windmill-app]'
reason: ContainersNotReady
status: "False"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2023-05-25T01:56:57Z"
message: 'containers with unready status: [windmill-app]'
reason: ContainersNotReady
status: "False"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2023-05-25T01:56:57Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: containerd://a9181c9213f5bf4b84151f2b7cc498ef3e6d510f4c80933827228e6c7f6faf87
image: ghcr.io/windmill-labs/windmill:1.103.0
imageID: ghcr.io/windmill-labs/windmill@sha256:8a7b6dec5b67834f0118260dfcd67212ad089806192cc0251bf245be1dac0ff5
lastState:
terminated:
containerID: containerd://a9181c9213f5bf4b84151f2b7cc498ef3e6d510f4c80933827228e6c7f6faf87
exitCode: 1
finishedAt: "2023-05-25T01:58:33Z"
reason: Error
startedAt: "2023-05-25T01:58:33Z"
name: windmill-app
ready: false
restartCount: 2
started: false
state:
waiting:
message: back-off 20s restarting failed container=windmill-app pod=windmill-app-5d9d88dff4-vxs8k_windmill(b609bee7-c6cc-4b75-a1e0-0c582292ece2)
reason: CrashLoopBackOff
conditions:
- lastProbeTime: null
lastTransitionTime: "2023-05-25T01:56:57Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2023-05-25T01:56:57Z"
message: 'containers with unready status: [windmill-app]'
reason: ContainersNotReady
status: "False"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2023-05-25T01:56:57Z"
message: 'containers with unready status: [windmill-app]'
reason: ContainersNotReady
status: "False"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2023-05-25T01:56:57Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: containerd://a9181c9213f5bf4b84151f2b7cc498ef3e6d510f4c80933827228e6c7f6faf87
image: ghcr.io/windmill-labs/windmill:1.103.0
imageID: ghcr.io/windmill-labs/windmill@sha256:8a7b6dec5b67834f0118260dfcd67212ad089806192cc0251bf245be1dac0ff5
lastState:
terminated:
containerID: containerd://a9181c9213f5bf4b84151f2b7cc498ef3e6d510f4c80933827228e6c7f6faf87
exitCode: 1
finishedAt: "2023-05-25T01:58:33Z"
reason: Error
startedAt: "2023-05-25T01:58:33Z"
name: windmill-app
ready: false
restartCount: 2
started: false
state:
waiting:
message: back-off 20s restarting failed container=windmill-app pod=windmill-app-5d9d88dff4-vxs8k_windmill(b609bee7-c6cc-4b75-a1e0-0c582292ece2)
reason: CrashLoopBackOff
No description
CookieMonster
CookieMonster13mo ago
i have the sidecar proxy connection to Google Cloud SQL running within the same pod as the windmill app. This should allow me to connect to the sql database using 127.0.0.1 but without more logging im kinda at a loss here is the deployed yaml output of the modified helm release
CookieMonster
CookieMonster13mo ago
essentially I am trying to connect this pod to google's SQL Cloud Service which is in another VPC. They offer a SQL auth proxy which you can run as a sidecar to the main application. But for the life of me I cannot figure out why the windmill application is crashing, and not offering any logging what so ever.
rubenf
rubenf13mo ago
Could you try making it work with psql instead of windmill And run a simple command
CookieMonster
CookieMonster13mo ago
not sure i quite follow. I was able to connect to the database using psql and google cloud shell. I am more curious why there are zero logs coming from the container itself :/
rubenf
rubenf13mo ago
did you set RUST_LOG=info ? One thing maybe is that the container never timeout on trying to establish the db connection but if the DATABASE_URL work for psql, it should work for windmill i'm sorry you're going through so much trouble
CookieMonster
CookieMonster13mo ago
yes its set in the config provided above ya its weird. just crashes with exit status code 1 and nothing else lol
rubenf
rubenf13mo ago
let me add one log before setting up the db connection and after it's done so we can know for sure it's hanging on the db connection
CookieMonster
CookieMonster13mo ago
❤️ I also will probably try and deploy another sample application to the cluster that connects to the database over google's sql proxy to just make sure.
rubenf
rubenf13mo ago
tracing::info!("Connecting to database...");
let db = windmill_common::connect_db(server_mode).await?;
tracing::info!("Database connected");
tracing::info!("Connecting to database...");
let db = windmill_common::connect_db(server_mode).await?;
tracing::info!("Database connected");
that is on latest (will be published soon)
CookieMonster
CookieMonster13mo ago
awesome. Let me know when its up, and ill pull down and take a look. Thank you so much @rubenf
rubenf
rubenf13mo ago
it's already up it only take a few mins to be published just pull latest
CookieMonster
CookieMonster13mo ago
sweet im on it! just getting through meetings
CookieMonster
CookieMonster13mo ago
so quick update, found out our test cluster was sending logs to a legacy stack driver. Sorry @rubenf. Good news tho is I have now logs. Did not see ur updated info log tho :/ Based on the loggin looks like I now need to seed the database IE perform the inital migration?
No description
rubenf
rubenf13mo ago
🙂 great news!
CookieMonster
CookieMonster13mo ago
working on this now