D
Dokploy8mo ago
max.d

System dies with 400 after a few hours

Logs show:
**⨯ PostgresError: the database system is in recovery mode
dokploy.1.9iig3pomqd5h@staging-bldry | at ErrorResponse (file:///app/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:788:26)
dokploy.1.9iig3pomqd5h@staging-bldry | at handle (file:///app/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:474:6)
dokploy.1.9iig3pomqd5h@staging-bldry | at Socket.data (file:///app/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:315:9)
dokploy.1.9iig3pomqd5h@staging-bldry | at Socket.emit (node:events:517:28)
dokploy.1.9iig3pomqd5h@staging-bldry | at addChunk (node:internal/streams/readable:368:12)
dokploy.1.9iig3pomqd5h@staging-bldry | at readableAddChunk (node:internal/streams/readable:341:9)
dokploy.1.9iig3pomqd5h@staging-bldry | at Readable.push (node:internal/streams/readable:278:10)
dokploy.1.9iig3pomqd5h@staging-bldry | at TCP.onStreamRead (node:internal/stream_base_commons:190:23)
dokploy.1.9iig3pomqd5h@staging-bldry | at TCP.callbackTrampoline (node:internal/async_hooks:128:17) {
dokploy.1.9iig3pomqd5h@staging-bldry | severity_local: 'FATAL',
dokploy.1.9iig3pomqd5h@staging-bldry | severity: 'FATAL',
dokploy.1.9iig3pomqd5h@staging-bldry | code: '57P03',
dokploy.1.9iig3pomqd5h@staging-bldry | file: 'postmaster.c',
dokploy.1.9iig3pomqd5h@staging-bldry | line: '2356',
dokploy.1.9iig3pomqd5h@staging-bldry | routine: 'ProcessStartupPacket'
dokploy.1.9iig3pomqd5h@staging-bldry | }
**⨯ PostgresError: the database system is in recovery mode
dokploy.1.9iig3pomqd5h@staging-bldry | at ErrorResponse (file:///app/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:788:26)
dokploy.1.9iig3pomqd5h@staging-bldry | at handle (file:///app/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:474:6)
dokploy.1.9iig3pomqd5h@staging-bldry | at Socket.data (file:///app/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:315:9)
dokploy.1.9iig3pomqd5h@staging-bldry | at Socket.emit (node:events:517:28)
dokploy.1.9iig3pomqd5h@staging-bldry | at addChunk (node:internal/streams/readable:368:12)
dokploy.1.9iig3pomqd5h@staging-bldry | at readableAddChunk (node:internal/streams/readable:341:9)
dokploy.1.9iig3pomqd5h@staging-bldry | at Readable.push (node:internal/streams/readable:278:10)
dokploy.1.9iig3pomqd5h@staging-bldry | at TCP.onStreamRead (node:internal/stream_base_commons:190:23)
dokploy.1.9iig3pomqd5h@staging-bldry | at TCP.callbackTrampoline (node:internal/async_hooks:128:17) {
dokploy.1.9iig3pomqd5h@staging-bldry | severity_local: 'FATAL',
dokploy.1.9iig3pomqd5h@staging-bldry | severity: 'FATAL',
dokploy.1.9iig3pomqd5h@staging-bldry | code: '57P03',
dokploy.1.9iig3pomqd5h@staging-bldry | file: 'postmaster.c',
dokploy.1.9iig3pomqd5h@staging-bldry | line: '2356',
dokploy.1.9iig3pomqd5h@staging-bldry | routine: 'ProcessStartupPacket'
dokploy.1.9iig3pomqd5h@staging-bldry | }
63 Replies
max.d
max.dOP8mo ago
GitHub
Dokploy BE dies after a few deploys · Issue #455 · Dokploy/dokploy
To Reproduce Idk run my deploys i guess Current vs. Expected behavior no crash Provide environment information last ubuntu on hetzner Which area(s) are affected? (Select all that apply) Application...
max.d
max.dOP8mo ago
out of disk space
Siumauricio
Siumauricio8mo ago
I’m going to say that 😅
max.d
max.dOP8mo ago
how can it run out of disk space after a few hours? looks like there needs to be an option to immediately prune old deploys especially failed images it consumed 40GB of space in a few hours
Siumauricio
Siumauricio8mo ago
You are doing so many deployments?
max.d
max.dOP8mo ago
i was testing runs yea
Siumauricio
Siumauricio8mo ago
Got it, you can activate the docker cleanup in the settings But that will run in everyday
max.d
max.dOP8mo ago
yeah thats what i thought
Siumauricio
Siumauricio8mo ago
If you are using the API you can hit the endpoint to cleanup Manually
max.d
max.dOP8mo ago
i have that active but it didnt get the chance to prune well i guess for a normal run scenario it's a non issue, but for debugs yea maybe mention in the docs that it goes out disk pronto
max.d
max.dOP8mo ago
ouch lol
No description
Siumauricio
Siumauricio8mo ago
You can do it trough the UI btw You have different options, clean unused images, unused containers, unused volumes clean builder & system prune
max.d
max.dOP8mo ago
dokploy.1.nkjq33ey3ooe@staging-bldry | Dokploy server is up and running!
dokploy.1.nkjq33ey3ooe@staging-bldry | Please wait for 15 seconds before opening the browser.
dokploy.1.nkjq33ey3ooe@staging-bldry | http://188.245.149.114:3000
dokploy.1.nkjq33ey3ooe@staging-bldry |
dokploy.1.nkjq33ey3ooe@staging-bldry |
dokploy.1.nkjq33ey3ooe@staging-bldry | Migration failed Error: getaddrinfo ENOTFOUND dokploy-postgres
dokploy.1.nkjq33ey3ooe@staging-bldry | at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:107:26) {
dokploy.1.nkjq33ey3ooe@staging-bldry | errno: -3008,
dokploy.1.nkjq33ey3ooe@staging-bldry | code: 'ENOTFOUND',
dokploy.1.nkjq33ey3ooe@staging-bldry | syscall: 'getaddrinfo',
dokploy.1.nkjq33ey3ooe@staging-bldry | hostname: 'dokploy-postgres'
dokploy.1.nkjq33ey3ooe@staging-bldry | }
dokploy.1.nkjq33ey3ooe@staging-bldry | Dokploy server is up and running!
dokploy.1.nkjq33ey3ooe@staging-bldry | Please wait for 15 seconds before opening the browser.
dokploy.1.nkjq33ey3ooe@staging-bldry | http://188.245.149.114:3000
dokploy.1.nkjq33ey3ooe@staging-bldry |
dokploy.1.nkjq33ey3ooe@staging-bldry |
dokploy.1.nkjq33ey3ooe@staging-bldry | Migration failed Error: getaddrinfo ENOTFOUND dokploy-postgres
dokploy.1.nkjq33ey3ooe@staging-bldry | at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:107:26) {
dokploy.1.nkjq33ey3ooe@staging-bldry | errno: -3008,
dokploy.1.nkjq33ey3ooe@staging-bldry | code: 'ENOTFOUND',
dokploy.1.nkjq33ey3ooe@staging-bldry | syscall: 'getaddrinfo',
dokploy.1.nkjq33ey3ooe@staging-bldry | hostname: 'dokploy-postgres'
dokploy.1.nkjq33ey3ooe@staging-bldry | }
looks like i messed it up, any idea how to reset it?
Siumauricio
Siumauricio8mo ago
You cannot access to the UI?
max.d
max.dOP8mo ago
nope ded
Siumauricio
Siumauricio8mo ago
Ok in that cases you can do this or you can do this docker service ls docker service scale postgres=1
max.d
max.dOP8mo ago
weird thing is its running
ID NAME MODE REPLICAS IMAGE PORTS
0d4ahcau2l4w bldry-api-91c604 replicated 1/1 bldry-api-91c604:latest
5sbej0m4xqc8 bldry-db-d66180 replicated 1/1 postgres:16.4
l2fltue0krhz dokploy replicated 1/1 dokploy/dokploy:latest
ay42ob9y98nl dokploy-postgres replicated 1/1 postgres:16
hjy1flvr9qcm dokploy-redis replicated 1/1 redis:7
cc5034zte81y dokploy-traefik replicated 1/1 traefik:v3.1.2
ID NAME MODE REPLICAS IMAGE PORTS
0d4ahcau2l4w bldry-api-91c604 replicated 1/1 bldry-api-91c604:latest
5sbej0m4xqc8 bldry-db-d66180 replicated 1/1 postgres:16.4
l2fltue0krhz dokploy replicated 1/1 dokploy/dokploy:latest
ay42ob9y98nl dokploy-postgres replicated 1/1 postgres:16
hjy1flvr9qcm dokploy-redis replicated 1/1 redis:7
cc5034zte81y dokploy-traefik replicated 1/1 traefik:v3.1.2
Siumauricio
Siumauricio8mo ago
ok do this, docker service scale dokploy=0 then docker service scale dokploy=1
max.d
max.dOP8mo ago
yep that did it
Siumauricio
Siumauricio8mo ago
nothing works? for me i can access http://188.245.149.114:3000/
max.d
max.dOP8mo ago
it started normally and immediately started re-running last builds
Siumauricio
Siumauricio8mo ago
Yes, this is a mechanism of the queues and bullmq that will retry if there was a server error, I think it would be ideal to remove this.
max.d
max.dOP8mo ago
yep should just start asap to avoid downtime i guess btw is there a way to scale to 3-4 instances without setting up container registry? just load balance between internal processes instead of multiple VPS node is single threaded i have some heavy ops that would lock up and i'm too lazy to make queues atm
Siumauricio
Siumauricio8mo ago
You can create multiple instances but in the same server Without setting up a registry In advanced -> replicas
max.d
max.dOP8mo ago
ui is a bit confusing then
No description
Siumauricio
Siumauricio8mo ago
We can improve for sure Did it worked ?
max.d
max.dOP8mo ago
nop
No description
max.d
max.dOP8mo ago
[{
"error": {
"json": {
"message": "insert or update on table \"application\" violates foreign key constraint \"application_registryId_registry_registryId_fk\"",
"code": -32603,
"data": {
"code": "INTERNAL_SERVER_ERROR",
"httpStatus": 500,
"path": "application.update",
"zodError": null
}
}
}
}]
[{
"error": {
"json": {
"message": "insert or update on table \"application\" violates foreign key constraint \"application_registryId_registry_registryId_fk\"",
"code": -32603,
"data": {
"code": "INTERNAL_SERVER_ERROR",
"httpStatus": 500,
"path": "application.update",
"zodError": null
}
}
}
}]
Siumauricio
Siumauricio8mo ago
🤔 What did you set?
max.d
max.dOP8mo ago
4
Siumauricio
Siumauricio8mo ago
Let me check Got it, there is a wrong validation in the frontend side
max.d
max.dOP8mo ago
error looks like drizzle foreign key constraint to me
Siumauricio
Siumauricio8mo ago
nop, is just because we have a wrong condition here
No description
Siumauricio
Siumauricio8mo ago
we are passing an empty registry id "" so that's why it throws
max.d
max.dOP8mo ago
ahhh fml gotcha ok i will have to wait for a fix then
Siumauricio
Siumauricio8mo ago
I pushed a new release in v0.8.2
max.d
max.dOP8mo ago
awesome, i just managed to secure my install so all ports are locked by default instead of having 3000 etc open 😄 will try to update via interface see what happens
max.d
max.dOP8mo ago
app tried to redeploy a working build and bricked itself, interesting
No description
max.d
max.dOP8mo ago
github provider died
No description
max.d
max.dOP8mo ago
huh
dokploy.1.3pnfje39gz58@staging-bldry | TypeError: fetch failed
dokploy.1.3pnfje39gz58@staging-bldry | at node:internal/deps/undici/undici:12618:11
dokploy.1.3pnfje39gz58@staging-bldry | at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
dokploy.1.3pnfje39gz58@staging-bldry | at async st (file:///app/dist/server.mjs:57:3111)
dokploy.1.3pnfje39gz58@staging-bldry | at async Ca (file:///app/dist/server.mjs:88:7652)
dokploy.1.3pnfje39gz58@staging-bldry | at async file:///app/dist/server.mjs:91:6462 {
dokploy.1.3pnfje39gz58@staging-bldry | cause: ConnectTimeoutError: Connect Timeout Error
dokploy.1.3pnfje39gz58@staging-bldry | at onConnectTimeout (node:internal/deps/undici/undici:7760:28)
dokploy.1.3pnfje39gz58@staging-bldry | at node:internal/deps/undici/undici:7716:50
dokploy.1.3pnfje39gz58@staging-bldry | at Immediate._onImmediate (node:internal/deps/undici/undici:7748:13)
dokploy.1.3pnfje39gz58@staging-bldry | at process.processImmediate (node:internal/timers:476:21) {
dokploy.1.3pnfje39gz58@staging-bldry | code: 'UND_ERR_CONNECT_TIMEOUT'
dokploy.1.3pnfje39gz58@staging-bldry | }
dokploy.1.3pnfje39gz58@staging-bldry | }
dokploy.1.3pnfje39gz58@staging-bldry | Server Started: 3000
dokploy.1.3pnfje39gz58@staging-bldry | TypeError: fetch failed
dokploy.1.3pnfje39gz58@staging-bldry | at node:internal/deps/undici/undici:12618:11
dokploy.1.3pnfje39gz58@staging-bldry | at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
dokploy.1.3pnfje39gz58@staging-bldry | at async st (file:///app/dist/server.mjs:57:3111)
dokploy.1.3pnfje39gz58@staging-bldry | at async Ca (file:///app/dist/server.mjs:88:7652)
dokploy.1.3pnfje39gz58@staging-bldry | at async file:///app/dist/server.mjs:91:6462 {
dokploy.1.3pnfje39gz58@staging-bldry | cause: ConnectTimeoutError: Connect Timeout Error
dokploy.1.3pnfje39gz58@staging-bldry | at onConnectTimeout (node:internal/deps/undici/undici:7760:28)
dokploy.1.3pnfje39gz58@staging-bldry | at node:internal/deps/undici/undici:7716:50
dokploy.1.3pnfje39gz58@staging-bldry | at Immediate._onImmediate (node:internal/deps/undici/undici:7748:13)
dokploy.1.3pnfje39gz58@staging-bldry | at process.processImmediate (node:internal/timers:476:21) {
dokploy.1.3pnfje39gz58@staging-bldry | code: 'UND_ERR_CONNECT_TIMEOUT'
dokploy.1.3pnfje39gz58@staging-bldry | }
dokploy.1.3pnfje39gz58@staging-bldry | }
dokploy.1.3pnfje39gz58@staging-bldry | Server Started: 3000
external connections should work fine weird ok enabling ufw no good, any idea how to close 3000?
Siumauricio
Siumauricio8mo ago
You closed the port 3000? But you can access to the domain?
max.d
max.dOP8mo ago
yeh i wanted to do this:
echo "{
\"iptables\": false
}" > /etc/docker/daemon.json

sed -i -e 's/DEFAULT_FORWARD_POLICY="DROP"/DEFAULT_FORWARD_POLICY="ACCEPT"/g' /etc/default/ufw
echo "{
\"iptables\": false
}" > /etc/docker/daemon.json

sed -i -e 's/DEFAULT_FORWARD_POLICY="DROP"/DEFAULT_FORWARD_POLICY="ACCEPT"/g' /etc/default/ufw
so that ufw blocks incoming connections to anything except 80 and 443 everyhthing works except github connection somehow stopped working
Siumauricio
Siumauricio8mo ago
What happen if you try to use git provider?
max.d
max.dOP8mo ago
sends to github ok but then redirect hangs
Siumauricio
Siumauricio8mo ago
I don’t know if the octokit library we use have some issues internally
max.d
max.dOP8mo ago
dokploy.1.3pnfje39gz58@staging-bldry | RequestError [HttpError]: Connect Timeout Error
dokploy.1.3pnfje39gz58@staging-bldry | at /app/node_modules/.pnpm/@[email protected]/node_modules/@octokit/request/dist-node/index.js:157:11
dokploy.1.3pnfje39gz58@staging-bldry | at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
dokploy.1.3pnfje39gz58@staging-bldry | at async requestWithGraphqlErrorHandling (/app/node_modules/.pnpm/@[email protected]_@[email protected]/node_modules/@octokit/plugin-retry/dist-node/index.js:71:20)
dokploy.1.3pnfje39gz58@staging-bldry | at async Job.doExecute (/app/node_modules/.pnpm/[email protected]/node_modules/bottleneck/light.js:405:18) {
dokploy.1.3pnfje39gz58@staging-bldry | status: 500,
dokploy.1.3pnfje39gz58@staging-bldry | request: {
dokploy.1.3pnfje39gz58@staging-bldry | method: 'POST',
dokploy.1.3pnfje39gz58@staging-bldry | url: 'https://api.github.com/app-manifests/3a02a6fa043cf14faab662b4a2a467691ad46fb4/conversions',
dokploy.1.3pnfje39gz58@staging-bldry | headers: {
dokploy.1.3pnfje39gz58@staging-bldry | accept: 'application/vnd.github.v3+json',
dokploy.1.3pnfje39gz58@staging-bldry | 'user-agent': 'octokit.js/3.1.2 octokit-core.js/5.2.0 Node.js/18.20.4 (linux; x64)'
dokploy.1.3pnfje39gz58@staging-bldry | },
dokploy.1.3pnfje39gz58@staging-bldry | request: {
dokploy.1.3pnfje39gz58@staging-bldry | hook: [Function: bound bound register],
dokploy.1.3pnfje39gz58@staging-bldry | retryCount: 3,
dokploy.1.3pnfje39gz58@staging-bldry | retries: 3,
dokploy.1.3pnfje39gz58@staging-bldry | retryAfter: 16
dokploy.1.3pnfje39gz58@staging-bldry | }
dokploy.1.3pnfje39gz58@staging-bldry | }
dokploy.1.3pnfje39gz58@staging-bldry | }
dokploy.1.3pnfje39gz58@staging-bldry | RequestError [HttpError]: Connect Timeout Error
dokploy.1.3pnfje39gz58@staging-bldry | at /app/node_modules/.pnpm/@[email protected]/node_modules/@octokit/request/dist-node/index.js:157:11
dokploy.1.3pnfje39gz58@staging-bldry | at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
dokploy.1.3pnfje39gz58@staging-bldry | at async requestWithGraphqlErrorHandling (/app/node_modules/.pnpm/@[email protected]_@[email protected]/node_modules/@octokit/plugin-retry/dist-node/index.js:71:20)
dokploy.1.3pnfje39gz58@staging-bldry | at async Job.doExecute (/app/node_modules/.pnpm/[email protected]/node_modules/bottleneck/light.js:405:18) {
dokploy.1.3pnfje39gz58@staging-bldry | status: 500,
dokploy.1.3pnfje39gz58@staging-bldry | request: {
dokploy.1.3pnfje39gz58@staging-bldry | method: 'POST',
dokploy.1.3pnfje39gz58@staging-bldry | url: 'https://api.github.com/app-manifests/3a02a6fa043cf14faab662b4a2a467691ad46fb4/conversions',
dokploy.1.3pnfje39gz58@staging-bldry | headers: {
dokploy.1.3pnfje39gz58@staging-bldry | accept: 'application/vnd.github.v3+json',
dokploy.1.3pnfje39gz58@staging-bldry | 'user-agent': 'octokit.js/3.1.2 octokit-core.js/5.2.0 Node.js/18.20.4 (linux; x64)'
dokploy.1.3pnfje39gz58@staging-bldry | },
dokploy.1.3pnfje39gz58@staging-bldry | request: {
dokploy.1.3pnfje39gz58@staging-bldry | hook: [Function: bound bound register],
dokploy.1.3pnfje39gz58@staging-bldry | retryCount: 3,
dokploy.1.3pnfje39gz58@staging-bldry | retries: 3,
dokploy.1.3pnfje39gz58@staging-bldry | retryAfter: 16
dokploy.1.3pnfje39gz58@staging-bldry | }
dokploy.1.3pnfje39gz58@staging-bldry | }
dokploy.1.3pnfje39gz58@staging-bldry | }
yeah it timed out, really weird because it should be able to connect fine disabled uwf -> fine
Siumauricio
Siumauricio8mo ago
Hmmm can you try the git provider? Also I have a question you setup the GitHub account when you was in ip:3000? Or when you was in the domain ?
max.d
max.dOP8mo ago
nah everything immediately via domain
Siumauricio
Siumauricio8mo ago
Strange, had no idea
max.d
max.dOP8mo ago
first thing after creating acc -> add domain
Siumauricio
Siumauricio8mo ago
Probably something internal of docker
max.d
max.dOP8mo ago
yeah i didnt like that it was open but disabling iptables breaks something internally
max.d
max.dOP8mo ago
GitHub
request.js/src/fetch-wrapper.ts at main · octokit/request.js
Send parameterized requests to GitHub’s APIs with sensible defaults in browsers and Node - octokit/request.js
max.d
max.dOP8mo ago
i would suggest terminate access by ip:3000 if domain is configured
Siumauricio
Siumauricio8mo ago
I don’t do that, because probably someone can touch the traefik config and break the traefik instance In that case you will lock the instance and you will cant access
max.d
max.dOP8mo ago
hm can still recover via ssh no?
ID NAME MODE REPLICAS IMAGE PORTS
0d4ahcau2l4w bldry-api-91c604 replicated 4/4 bldry-api-91c604:latest
ID NAME MODE REPLICAS IMAGE PORTS
0d4ahcau2l4w bldry-api-91c604 replicated 4/4 bldry-api-91c604:latest
worked btw!
Siumauricio
Siumauricio8mo ago
Yep, but you probably need to tweak the traefik files and maybe running some docker commands
max.d
max.dOP8mo ago
Stack Overflow
What is the best practice of docker + ufw under Ubuntu
I just tried out Docker. It is awesome but seems not work nicely with ufw. By default, docker will manipulate the iptables a little bit. The outcome is not a bug but not what I expected. For more d...
Siumauricio
Siumauricio8mo ago
so you blocked everything except the UI trough the domain?
max.d
max.dOP8mo ago
yes, though not sure if a bug but github app was created like this
No description
max.d
max.dOP8mo ago
i reconfigured ufw so it needs to allow external connections to docker, otherwise docker has acces to the internet but not other way around
max.d
max.dOP8mo ago
and back online, now works
No description
max.d
max.dOP8mo ago
i can now confirm the app was created without domain, with domain works perfect

Did you find this page helpful?