Is anyone else having issues with Supabase us-east-1?

I'm getting transient issues when querying the database via the Supabase JS SDK or performing auth queries.

42 Replies

EMPZOP•8mo ago

Some requests are taking a lot and then failing with server error.

No one else reporting anything here or github... No status alert from SB. My east instance is working OK from dashboard and REST calls with supabase-js

vick•8mo ago

My instances on east-1 are working just fine, in that my web app is able to use Supabase. Did not try logging into the console.

EMPZOP•8mo ago

My console says everything is fine. I have 2 projects on my org (both on us-east-1). One is dev (I hit this one from my local machine) and another is for prod (vercel hits it). Both fail every now and then. It's been like that since yesterday.

EMPZOP•8mo ago

I'm on the Pro plan and my instances are barely using their available resources.

garyaustin•8mo ago

Go to this and see if any of those are out of whack. Check your Query Performance report in the DB. Any Postgres log errors? What are your failure errors? Any code changes on your part yesterday? It seems odd you would have two instances acting up.

EMPZOP•8mo ago

Nope, everything looks just fine.

garyaustin•8mo ago

Nothing on other questions? Unless you provide more info/errors/logs, something I don't think you will get more help from users here. Still no reports here or github or status of anyone else having issues. You could try support as they can see your instances, but if true only your two instances are doing this and you are accessing it two different ways with the same code, there is one common denominator.

EMPZOP•8mo ago

I'l get you more info soon, need to finish some stuff. I'm developing locally against one of the instances and sometimes I get this error in the console (Next.js app) and I get logged out of my app. I've tested my internet connection and it's fine.

 [TypeError: fetch failed] {
  cause:  [ConnectTimeoutError: Connect Timeout Error] {
  name: 'ConnectTimeoutError',
  code: 'UND_ERR_CONNECT_TIMEOUT',
  message: 'Connect Timeout Error'
}

 [TypeError: fetch failed] {
  cause:  [ConnectTimeoutError: Connect Timeout Error] {
  name: 'ConnectTimeoutError',
  code: 'UND_ERR_CONNECT_TIMEOUT',
  message: 'Connect Timeout Error'
}

TypeError: fetch failed
    at node:internal/deps/undici/undici:13392:13
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async _handleRequest (webpack-internal:///(rsc)/./node_modules/@supabase/auth-js/dist/module/lib/fetch.js:115:18)
    at async _request (webpack-internal:///(rsc)/./node_modules/@supabase/auth-js/dist/module/lib/fetch.js:105:18)
    at async eval (webpack-internal:///(rsc)/./node_modules/@supabase/auth-js/dist/module/GoTrueClient.js:905:24)
    at async SupabaseAuthClient._useSession (webpack-internal:///(rsc)/./node_modules/@supabase/auth-js/dist/module/GoTrueClient.js:805:20)
    at async SupabaseAuthClient._getUser (webpack-internal:///(rsc)/./node_modules/@supabase/auth-js/dist/module/GoTrueClient.js:895:20)
    at async eval (webpack-internal:///(rsc)/./node_modules/@supabase/auth-js/dist/module/GoTrueClient.js:882:20)
    at async eval (webpack-internal:///(rsc)/./node_modules/@supabase/auth-js/dist/module/GoTrueClient.js:750:28) {
  [cause]: ConnectTimeoutError: Connect Timeout Error (attempted addresses: 172.64.149.246:443, timeout: 10000ms)
      at onConnectTimeout (node:internal/deps/undici/undici:2599:28)
      at Immediate._onImmediate (node:internal/deps/undici/undici:2568:35)
      at process.processImmediate (node:internal/timers:483:21)
      at process.callbackTrampoline (node:internal/async_hooks:130:17) {
    code: 'UND_ERR_CONNECT_TIMEOUT'
  }
}

TypeError: fetch failed
    at node:internal/deps/undici/undici:13392:13
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async _handleRequest (webpack-internal:///(rsc)/./node_modules/@supabase/auth-js/dist/module/lib/fetch.js:115:18)
    at async _request (webpack-internal:///(rsc)/./node_modules/@supabase/auth-js/dist/module/lib/fetch.js:105:18)
    at async eval (webpack-internal:///(rsc)/./node_modules/@supabase/auth-js/dist/module/GoTrueClient.js:905:24)
    at async SupabaseAuthClient._useSession (webpack-internal:///(rsc)/./node_modules/@supabase/auth-js/dist/module/GoTrueClient.js:805:20)
    at async SupabaseAuthClient._getUser (webpack-internal:///(rsc)/./node_modules/@supabase/auth-js/dist/module/GoTrueClient.js:895:20)
    at async eval (webpack-internal:///(rsc)/./node_modules/@supabase/auth-js/dist/module/GoTrueClient.js:882:20)
    at async eval (webpack-internal:///(rsc)/./node_modules/@supabase/auth-js/dist/module/GoTrueClient.js:750:28) {
  [cause]: ConnectTimeoutError: Connect Timeout Error (attempted addresses: 172.64.149.246:443, timeout: 10000ms)
      at onConnectTimeout (node:internal/deps/undici/undici:2599:28)
      at Immediate._onImmediate (node:internal/deps/undici/undici:2568:35)
      at process.processImmediate (node:internal/timers:483:21)
      at process.callbackTrampoline (node:internal/async_hooks:130:17) {
    code: 'UND_ERR_CONNECT_TIMEOUT'
  }
}

garyaustin•8mo ago

That appears to be from Cloudflare, but could just be a timeout... not sure Check the API Gateway log to see if it gets there.
Check the Auth log. It mentions getUser in it.

EMPZOP•8mo ago

Is that ip addres ssome kind of global load balancer of Supabase? Because I'm pinging that and I get a 15ms latency. I'm in Spain so I should not have such low latency against a US east database.

garyaustin•8mo ago

It is not a load balancer but a CDN and DDOS protection. If it does not get a response it will timeout.

EMPZOP•8mo ago

API gateway shows all fine, I see my requests and none have failed, even though I had the issue several times in the past few hours I've been developing. Looks like it fails before reaching the API gateway

garyaustin•8mo ago

But does that API request get there?

EMPZOP•8mo ago

Well, it's hard to know A lot of requests are made every time I refresh a page or something so, hard to tell when it fails if it reached the API gateway or failed before.

garyaustin•8mo ago

It would be the /user endpoint... but depending on how your code is written you could be doing several of them.

EMPZOP•8mo ago

I see some of these

garyaustin•8mo ago

I'm going by that... So you should see a auth/v1/user request.

EMPZOP•8mo ago

Yeap, there are multiple auth/v1/user requests

EMPZOP•8mo ago

These are weird.

EMPZOP•8mo ago

I have not logged out at all. No idea why those auth/v1/token are returning 400.

garyaustin•8mo ago

404 on storage could also be you are not meeting RLS because you are not signed in. Did you change your auth code yesterday?

EMPZOP•8mo ago

Not at all.

garyaustin•8mo ago

Have you restarted the instance? Are you expecting all those 404 for the files?

EMPZOP•8mo ago

No, I have not. Gonna do that now and also my computer because you never know... No, I should have access to those if I'm logged in.

garyaustin•8mo ago

The last update to supabase-js (if you always use latest) was last week.
Did you upgrade your infrastructure to a new release of Postgres/Auth yesterday?

EMPZOP•8mo ago

I upgraded supabase-js a few days ago. But I have not updated supabase-ssr for example (I'm still on 0.4.1).

garyaustin•8mo ago

Hard to tell if a failing getUser (the timeout) starts this or if that is a symptom. But it looks like you are not signed in (which still should not cause a timeout). Are you simulating a load with your local/dev or is this all from one user?

EMPZOP•8mo ago

It's just me. I've been working with this instance and this Next.js project for several months now. This is the first time I'm experiencing these weird issues during development. But I'm also seeing some issues on my prod instance. Like a query failing multiple times inside an Upstash workflow (which retries 3 times with a lot of backoff) It just doesn't make sence that a Vercel function calling a Supabase REST endpoint failing 3 consecutive times with minutes in-between.

garyaustin•8mo ago

You could try and isolate that maybe and put in a support request. This auth stuff would be very difficult to point to your instance versus your code. So much interaction going on. Also no one else having issues at least publicly.

EMPZOP•8mo ago

Vercel runs on AWS and Supabase also runs on AWS, right? If my Vercel functions run on us-east-1 and my Supabase instance is on us-east-1, that should work as optimal as we can get, right?

garyaustin•8mo ago

I thought vercel functions ran close to the user. But not something I've used.

EMPZOP•8mo ago

Edge functions do that. Regular serverless function are fixed to a single location.

garyaustin•8mo ago

K Even if scattered you should not be timing out as queries can run many seconds.

EMPZOP•8mo ago

    at node:internal/deps/undici/undici:13392:13
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at runNextTicks (node:internal/process/task_queues:64:3)
    at process.processImmediate (node:internal/timers:454:9)
    at process.callbackTrampoline (node:internal/async_hooks:130:17) {
  [cause]: ConnectTimeoutError: Connect Timeout Error (attempted addresses: 104.18.38.10:443, timeout: 10000ms)
      at onConnectTimeout (node:internal/deps/undici/undici:2599:28)
      at Immediate._onImmediate (node:internal/deps/undici/undici:2568:35)
      at process.processImmediate (node:internal/timers:483:21)
      at process.callbackTrampoline (node:internal/async_hooks:130:17) {
    code: 'UND_ERR_CONNECT_TIMEOUT'
  }
}
TypeError: fetch failed
    at node:internal/deps/undici/undici:13392:13
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at runNextTicks (node:internal/process/task_queues:64:3)
    at process.processImmediate (node:internal/timers:454:9)
    at process.callbackTrampoline (node:internal/async_hooks:130:17) {
  [cause]: ConnectTimeoutError: Connect Timeout Error (attempted addresses: 104.18.38.10:443, timeout: 10000ms)
      at onConnectTimeout (node:internal/deps/undici/undici:2599:28)
      at Immediate._onImmediate (node:internal/deps/undici/undici:2568:35)
      at process.processImmediate (node:internal/timers:483:21)
      at process.callbackTrampoline (node:internal/async_hooks:130:17) {
    code: 'UND_ERR_CONNECT_TIMEOUT'
  }
}

    at node:internal/deps/undici/undici:13392:13
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at runNextTicks (node:internal/process/task_queues:64:3)
    at process.processImmediate (node:internal/timers:454:9)
    at process.callbackTrampoline (node:internal/async_hooks:130:17) {
  [cause]: ConnectTimeoutError: Connect Timeout Error (attempted addresses: 104.18.38.10:443, timeout: 10000ms)
      at onConnectTimeout (node:internal/deps/undici/undici:2599:28)
      at Immediate._onImmediate (node:internal/deps/undici/undici:2568:35)
      at process.processImmediate (node:internal/timers:483:21)
      at process.callbackTrampoline (node:internal/async_hooks:130:17) {
    code: 'UND_ERR_CONNECT_TIMEOUT'
  }
}
TypeError: fetch failed
    at node:internal/deps/undici/undici:13392:13
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at runNextTicks (node:internal/process/task_queues:64:3)
    at process.processImmediate (node:internal/timers:454:9)
    at process.callbackTrampoline (node:internal/async_hooks:130:17) {
  [cause]: ConnectTimeoutError: Connect Timeout Error (attempted addresses: 104.18.38.10:443, timeout: 10000ms)
      at onConnectTimeout (node:internal/deps/undici/undici:2599:28)
      at Immediate._onImmediate (node:internal/deps/undici/undici:2568:35)
      at process.processImmediate (node:internal/timers:483:21)
      at process.callbackTrampoline (node:internal/async_hooks:130:17) {
    code: 'UND_ERR_CONNECT_TIMEOUT'
  }
}

I'm switching back to the main branch (that's on prod) from the branch I've been working on since yesterday to see if I get the same weird behaviour.

garyaustin•8mo ago

That trace is hard to know what it is. I don't see anything Supabase specific. The IP is also cloudflare, but maybe Vercel uses that too.

EMPZOP•8mo ago

Well, doesn't seem to happen on main. Very weird, all the changes are just deeper level and silly components. Nothing that should cause timeouts or getting logged out. Anyway, thanks for the help.

garyaustin•8mo ago

Do you use onAuthStateChange event handler in any of these functions?

EMPZOP•8mo ago

Nope

garyaustin•8mo ago

K. There is a hang condition that can occur if not done correctly with another supabase call... like getUser().

EMPZOP•8mo ago

Well, I had no luck with any of this. I'm still facing this issue locally even after switching to my main branch which is what I have deployed on production. Tried multiple versions of supase-js and ssr. Multiple version of Node (20 and 22). My server console will just randomly log this error and log me out.

TypeError: fetch failed
    at node:internal/deps/undici/undici:13502:13
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async _handleRequest (webpack-internal:///(rsc)/./node_modules/@supabase/auth-js/dist/module/lib/fetch.js:115:18)
    at async _request (webpack-internal:///(rsc)/./node_modules/@supabase/auth-js/dist/module/lib/fetch.js:105:18)
    at async eval (webpack-internal:///(rsc)/./node_modules/@supabase/auth-js/dist/module/GoTrueClient.js:905:24)
    at async SupabaseAuthClient._useSession (webpack-internal:///(rsc)/./node_modules/@supabase/auth-js/dist/module/GoTrueClient.js:805:20)
    at async SupabaseAuthClient._getUser (webpack-internal:///(rsc)/./node_modules/@supabase/auth-js/dist/module/GoTrueClient.js:895:20)
    at async eval (webpack-internal:///(rsc)/./node_modules/@supabase/auth-js/dist/module/GoTrueClient.js:882:20)
    at async eval (webpack-internal:///(rsc)/./node_modules/@supabase/auth-js/dist/module/GoTrueClient.js:750:28) {
  [cause]: ConnectTimeoutError: Connect Timeout Error (attempted addresses: 104.18.38.10:443, timeout: 10000ms)
      at onConnectTimeout (node:internal/deps/undici/undici:2602:28)
      at Immediate._onImmediate (node:internal/deps/undici/undici:2568:35)
      at process.processImmediate (node:internal/timers:491:21)
      at process.callbackTrampoline (node:internal/async_hooks:130:17) {
    code: 'UND_ERR_CONNECT_TIMEOUT'
  }
}

TypeError: fetch failed
    at node:internal/deps/undici/undici:13502:13
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async _handleRequest (webpack-internal:///(rsc)/./node_modules/@supabase/auth-js/dist/module/lib/fetch.js:115:18)
    at async _request (webpack-internal:///(rsc)/./node_modules/@supabase/auth-js/dist/module/lib/fetch.js:105:18)
    at async eval (webpack-internal:///(rsc)/./node_modules/@supabase/auth-js/dist/module/GoTrueClient.js:905:24)
    at async SupabaseAuthClient._useSession (webpack-internal:///(rsc)/./node_modules/@supabase/auth-js/dist/module/GoTrueClient.js:805:20)
    at async SupabaseAuthClient._getUser (webpack-internal:///(rsc)/./node_modules/@supabase/auth-js/dist/module/GoTrueClient.js:895:20)
    at async eval (webpack-internal:///(rsc)/./node_modules/@supabase/auth-js/dist/module/GoTrueClient.js:882:20)
    at async eval (webpack-internal:///(rsc)/./node_modules/@supabase/auth-js/dist/module/GoTrueClient.js:750:28) {
  [cause]: ConnectTimeoutError: Connect Timeout Error (attempted addresses: 104.18.38.10:443, timeout: 10000ms)
      at onConnectTimeout (node:internal/deps/undici/undici:2602:28)
      at Immediate._onImmediate (node:internal/deps/undici/undici:2568:35)
      at process.processImmediate (node:internal/timers:491:21)
      at process.callbackTrampoline (node:internal/async_hooks:130:17) {
    code: 'UND_ERR_CONNECT_TIMEOUT'
  }
}

I'm pretty sure my 1 Gigabit connection is just fine...

garyaustin•8mo ago

Need to somehow figure out why that getUser call is "hanging". If it is getting to Supabase or hanging the Vercel function (if that is where it is running). Note: https://vercel.com/guides/what-can-i-do-about-vercel-serverless-functions-timing-out Has a default 10 second timeout. Supabase and cloudflare does not have a 10 second timeout. But getUser should never take more than a second if that.

Gaming

Programming

Is anyone else having issues with Supabase us-east-1?

Did you find this page helpful?