Twenty•3mo ago

Upgrade failed - is my deployment broken now? Can I fix it?

I fear I may have broken my twenty deployment. Here is what I did: I thought I was on v0.55.1 so I just did 1. dbdump 2. podman-compose down 3. Edit .env file setting TAG=0.55 (was TAG=latest before) 4. podman-compose up Now, twenty got up immediately but it did not download new images. So I did the following: * podman-compose down * podman-compose pull Now it downloaded new images. I started twenty: * podman-compose up -d Now twenty starts up. But I cannot login. When I enter a username, I get this error: QueryFailedError I then tried inspecting the database dump. And now it seems that I may have been on 0.51.0 and not 0.55.1 as I thought.... So have I completely broken my deployment? OR can I fix this? If so, how?

130 Replies

Prastoin•3mo ago

Hello @EbenezerIbiza I would recommend recovering from a 0.51 and proceed to an incremental upgrade up to 0.55

EbenezerIbizaOP•3mo ago

THanks for your reply. So do I just change TAG=0.51 in the .env and then do podman-compose pull again and bring back up? And then restore the database and proceed with incremental upgrade? @prastoin so I tried doing this: * stopped twenty * changed TAG in .env to 0.51 * did podman-compose pull * Then broought up twenty using podman-compose up -d * Then tried to login. I Then get this error:

column User__User_workspaces.defaultAvatarUrl does not exist

column User__User_workspaces.defaultAvatarUrl does not exist

I did a database dump again now and diffed with the old one. This is the diff:

--- databases_backup2.sql       2025-06-23 14:24:46.842963554 +0200
+++ databases_backup.sql        2025-06-23 13:01:44.717746982 +0200
@@ -39,8 +39,8 @@


 SET lock_timeout = 0;
@@ -67,8 +67,8 @@


 SET lock_timeout = 0;
@@ -93,8 +93,8 @@

 SET statement_timeout = 0;
 SET lock_timeout = 0;
@@ -2262,8 +2262,7 @@
 79**********************************   b8**********************************    2025-06-16 19:22:30.271+00      \N      2025-04-17 20:10:59.942+00      2025-04-17 19:22:30.271498+00   2025-04-17 20:10:59.943056+00   10f069ce-793a-4ada-b841-0bb11359b874    REFRESH_TOKEN   \N      \N
 84**********************************   b8**********************************    2025-06-16 20:10:59.968+00      \N      2025-04-19 10:54:46.414+00      2025-04-17 20:10:59.969011+00   2025-04-19 10:54:46.415111+00   10f069ce-793a-4ada-b841-0bb11359b874    REFRESH_TOKEN   \N      \N
 7d**********************************   b8**********************************    2025-06-18 10:54:46.492+00      \N      \N      2025-04-19 10:54:46.492456+00   2025-04-19 10:54:46.492456+00   10f069ce-793a-4ada-b841-0bb11359b874    REFRESH_TOKEN   \N      \N
-2f**********************************   b8**********************************    2025-08-15 17:39:44.456+00      \N      2025-06-23 11:05:36.889+00      2025-06-16 17:39:44.458631+00   2025-06-23 11:05:36.892566+00   10f069ce-793a-4ada-b841-0bb11359b874    REFRESH_TOKEN   \N      \N
-a7**********************************   b8**********************************    2025-08-22 11:05:36.975+00      \N      2025-06-23 12:21:53.422+00      2025-06-23 11:05:36.977769+00   2025-06-23 12:21:53.422569+00   10f069ce-793a-4ada-b841-0bb11359b874    REFRESH_TOKEN   \N      \N
+2f**********************************   b8**********************************    2025-08-15 17:39:44.456+00      \N      \N      2025-06-16 17:39:44.458631+00   2025-06-16 17:39:44.458631+00   10f069ce-793a-4ada-b841-0bb11359b874    REFRESH_TOKEN   \N      \N
 \.

--- databases_backup2.sql       2025-06-23 14:24:46.842963554 +0200
+++ databases_backup.sql        2025-06-23 13:01:44.717746982 +0200
@@ -39,8 +39,8 @@


 SET lock_timeout = 0;
@@ -67,8 +67,8 @@


 SET lock_timeout = 0;
@@ -93,8 +93,8 @@

 SET statement_timeout = 0;
 SET lock_timeout = 0;
@@ -2262,8 +2262,7 @@
 79**********************************   b8**********************************    2025-06-16 19:22:30.271+00      \N      2025-04-17 20:10:59.942+00      2025-04-17 19:22:30.271498+00   2025-04-17 20:10:59.943056+00   10f069ce-793a-4ada-b841-0bb11359b874    REFRESH_TOKEN   \N      \N
 84**********************************   b8**********************************    2025-06-16 20:10:59.968+00      \N      2025-04-19 10:54:46.414+00      2025-04-17 20:10:59.969011+00   2025-04-19 10:54:46.415111+00   10f069ce-793a-4ada-b841-0bb11359b874    REFRESH_TOKEN   \N      \N
 7d**********************************   b8**********************************    2025-06-18 10:54:46.492+00      \N      \N      2025-04-19 10:54:46.492456+00   2025-04-19 10:54:46.492456+00   10f069ce-793a-4ada-b841-0bb11359b874    REFRESH_TOKEN   \N      \N
-2f**********************************   b8**********************************    2025-08-15 17:39:44.456+00      \N      2025-06-23 11:05:36.889+00      2025-06-16 17:39:44.458631+00   2025-06-23 11:05:36.892566+00   10f069ce-793a-4ada-b841-0bb11359b874    REFRESH_TOKEN   \N      \N
-a7**********************************   b8**********************************    2025-08-22 11:05:36.975+00      \N      2025-06-23 12:21:53.422+00      2025-06-23 11:05:36.977769+00   2025-06-23 12:21:53.422569+00   10f069ce-793a-4ada-b841-0bb11359b874    REFRESH_TOKEN   \N      \N
+2f**********************************   b8**********************************    2025-08-15 17:39:44.456+00      \N      \N      2025-06-16 17:39:44.458631+00   2025-06-16 17:39:44.458631+00   10f069ce-793a-4ada-b841-0bb11359b874    REFRESH_TOKEN   \N      \N
 \.

Do I need to restore the original db dump? Or is my issue unrelated to this? On further inspection, it seems to me none of these differences have any major relevance. The first ones are just comments (I edited those out). The last ones are real differences, but they seem to only be some app tokens....

Prastoin•3mo ago

You need to restore a 0.51 backup of your database while changing the TAG to 0.51

EbenezerIbizaOP•3mo ago

@prastoin : I think this is what I did. So: before I did the upgrade earlier. That is, before I changed anything and while I had a working twenty 0.51 instance, I did a database backup using pg_dump. I have saved that sql file. Then I did the upgrade/pull and got the 0.55 and broke my deployment. NOW what I have done is: podman-compose down Went into my .env and changed TAG to TAG=0.51 podman-compose pull podman-compose up -d Do you mean I have to recover the original backup into my database now?

Prastoin•3mo ago

Indeed you need to restore the dump 0.51 when being in the 0.51 version Then start incrementally upgrading to 0.55

EbenezerIbizaOP•3mo ago

Right. Will try that.

EbenezerIbizaOP•3mo ago

Wow. That failed spectacularly: https://privatebin.net/?643f0aecd9f424a7#FQ29PUfZnVhDMjqavz2X5GwZRgNnJeBgdSs7x1hbbZxn

PrivateBin

Encrypted note on PrivateBin

Visit this link to see the note. Giving the URL to anyone allows them to access the note, too.

EbenezerIbizaOP•3mo ago

Lots of errors. I tried following instructions to restore the dump from the Upgrade instructions: on https://twenty.com/developers/section/self-hosting/upgrade-guide I guess the dump has to delete the db objects before loading?

Prastoin•3mo ago

Please drop your existing database before restoring

EbenezerIbizaOP•3mo ago

So just run: podman exec -i twenty-db psql -U twenty Then drop database twenty ?? I did DROP DATABASE twenty; CREATE DATABASE twenty; Then ran this: cat databases_backup.sql | time podman exec -i twenty-db psql -U twenty I now brought back up twenty... Then try to login. I get this error: column User__User_workspaces.defaultAvatarUrl does not exist

Prastoin•3mo ago

You need to drop twenty database before restoring it If you don't you will face conflicts as you just did

EbenezerIbizaOP•3mo ago

@prastoin so, if I drop the db, I get this error when trying to load: $ cat databases_backup.sql | time podman exec -i twenty-db psql -U twenty psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: database "twenty" does not exist podman exec -i twenty-db psql -U twenty 0,04s user 0,02s system 44% cpu 0,153 total IF I create the database and then run it, then I will end up with an empty databse.... This is strange because the dump seems to have this line: CREATE DATABASE twenty WITH TEMPLATE = template0 ENCODING = 'UTF8' LOCALE_PROVIDER = libc LOCALE = 'en_US.utf8'; To create the db

Prastoin•3mo ago

IF I create the database and then run it, then I will end up with an empty databse....

Any errors in logs ?

EbenezerIbizaOP•3mo ago

This is the full output: $ cat databases_backup.sql | time podman exec -i twenty-db psql -U twenty psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: database "twenty" does not exist podman exec -i twenty-db psql -U twenty 0,04s user 0,03s system 33% cpu 0,192 total From the restore. Not sure what logs to look for

Prastoin•3mo ago

Please create an empty database twenty

EbenezerIbizaOP•3mo ago

emplate1=# CREATE DATABASE twenty WITH TEMPLATE = template0 ENCODING = 'UTF8' LOCALE_PROVIDER = libc LOCALE = 'en_US.utf8';
CREATE DATABASE



Then I run:

$ cat databases_backup.sql | time podman exec -i twenty-db psql -U twenty



It MOSTLY goes fine. But there is ONE error:

ERROR:  role "twenty" already exists


All the others seem to go fine. (lots of SET, CREATE, etc.)

emplate1=# CREATE DATABASE twenty WITH TEMPLATE = template0 ENCODING = 'UTF8' LOCALE_PROVIDER = libc LOCALE = 'en_US.utf8';
CREATE DATABASE



Then I run:

$ cat databases_backup.sql | time podman exec -i twenty-db psql -U twenty



It MOSTLY goes fine. But there is ONE error:

ERROR:  role "twenty" already exists


All the others seem to go fine. (lots of SET, CREATE, etc.)

I mean, I can delete that role... But then I will not be able to connect and do the restore.... So I guess this is a bootstrap problem... This is how it looks after I have loaded the backup: $ podman exec -it twenty-db psql -U twenty -d template1 psql (16.9 (Debian 16.9-1.pgdg120+1)) Type "help" for help. template1=# \c twenty You are now connected to database "twenty" as user "twenty". twenty=# \d Did not find any relations. twenty=# \dt Did not find any relations. twenty=# \z Access privileges Schema | Name | Type | Access privileges | Column privileges | Policies --------+------+------+-------------------+-------------------+---------- (0 rows) twenty=#

Prastoin•3mo ago

You can edit your dump file manually for not having to create the role Or adapt your psql/pg_restore command in order to handle conflicts smoother

EbenezerIbizaOP•3mo ago

OK. I edit the dump file and: commented out the CREATE ROLE statement and the CREATE DATABASE statement; Then I DROPped the twenty database and recreated it using the CREATE DATABASE statement from the script. Then re-ran the restore. It went without any problems. I have now restarted twenty. I then go do twenty in my browser. When I try to login, I get this error: column User__User_workspaces.defaultAvatarUrl does not exist When I connect to the database, it seems to be empty:

$ podman exec -it twenty-db psql -U twenty -d twenty
psql (16.9 (Debian 16.9-1.pgdg120+1))
Type "help" for help.

twenty=# \d
Did not find any relations.
twenty=# \dt
Did not find any relations.
twenty=# \l
                                                   List of databases
   Name    | Owner  | Encoding | Locale Provider |  Collate   |   Ctype    | ICU Locale | ICU Rules | Access privileges
-----------+--------+----------+-----------------+------------+------------+------------+-----------+-------------------
 postgres  | twenty | UTF8     | libc            | en_US.utf8 | en_US.utf8 |            |           |
 template0 | twenty | UTF8     | libc            | en_US.utf8 | en_US.utf8 |            |           | =c/twenty        +
           |        |          |                 |            |            |            |           | twenty=CTc/twenty
 template1 | twenty | UTF8     | libc            | en_US.utf8 | en_US.utf8 |            |           | =c/twenty        +
           |        |          |                 |            |            |            |           | twenty=CTc/twenty
 twenty    | twenty | UTF8     | libc            | en_US.utf8 | en_US.utf8 |            |           |
(4 rows)

twenty=#

$ podman exec -it twenty-db psql -U twenty -d twenty
psql (16.9 (Debian 16.9-1.pgdg120+1))
Type "help" for help.

twenty=# \d
Did not find any relations.
twenty=# \dt
Did not find any relations.
twenty=# \l
                                                   List of databases
   Name    | Owner  | Encoding | Locale Provider |  Collate   |   Ctype    | ICU Locale | ICU Rules | Access privileges
-----------+--------+----------+-----------------+------------+------------+------------+-----------+-------------------
 postgres  | twenty | UTF8     | libc            | en_US.utf8 | en_US.utf8 |            |           |
 template0 | twenty | UTF8     | libc            | en_US.utf8 | en_US.utf8 |            |           | =c/twenty        +
           |        |          |                 |            |            |            |           | twenty=CTc/twenty
 template1 | twenty | UTF8     | libc            | en_US.utf8 | en_US.utf8 |            |           | =c/twenty        +
           |        |          |                 |            |            |            |           | twenty=CTc/twenty
 twenty    | twenty | UTF8     | libc            | en_US.utf8 | en_US.utf8 |            |           |
(4 rows)

twenty=#

This is strange because the recovery did load the dump. But not sure where it loaded it to then.... Ah! It is NOT empty. It is just not in the public schema:

twenty=# \dn
                     List of schemas
                Name                 |       Owner
-------------------------------------+-------------------
 core                                | twenty
 metadata                            | twenty
 public                              | pg_database_owner
 workspace_103occrfmgm190gf6l4li4ub8 | twenty
(4 rows)

twenty=# \dn+
                                                      List of schemas
                Name                 |       Owner       |           Access privileges            |      Description
-------------------------------------+-------------------+----------------------------------------+------------------------
 core                                | twenty            |                                        |
 metadata                            | twenty            |                                        |
 public                              | pg_database_owner | pg_database_owner=UC/pg_database_owner+| standard public schema
                                     |                   | =U/pg_database_owner                   |
 workspace_103occrfmgm190gf6l4li4ub8 | twenty            |                                        |
(4 rows)

twenty=#

twenty=# \dn
                     List of schemas
                Name                 |       Owner
-------------------------------------+-------------------
 core                                | twenty
 metadata                            | twenty
 public                              | pg_database_owner
 workspace_103occrfmgm190gf6l4li4ub8 | twenty
(4 rows)

twenty=# \dn+
                                                      List of schemas
                Name                 |       Owner       |           Access privileges            |      Description
-------------------------------------+-------------------+----------------------------------------+------------------------
 core                                | twenty            |                                        |
 metadata                            | twenty            |                                        |
 public                              | pg_database_owner | pg_database_owner=UC/pg_database_owner+| standard public schema
                                     |                   | =U/pg_database_owner                   |
 workspace_103occrfmgm190gf6l4li4ub8 | twenty            |                                        |
(4 rows)

twenty=#

It seems that the column that it complains about not existing is indeed existing:

twenty=# SELECT table_schema,
       table_name,
       column_name
FROM information_schema.columns
WHERE column_name = 'defaultAvatarUrl';
 table_schema | table_name |   column_name
--------------+------------+------------------
 core         | user       | defaultAvatarUrl
(1 row)

twenty=#

twenty=# SELECT table_schema,
       table_name,
       column_name
FROM information_schema.columns
WHERE column_name = 'defaultAvatarUrl';
 table_schema | table_name |   column_name
--------------+------------+------------------
 core         | user       | defaultAvatarUrl
(1 row)

twenty=#

EbenezerIbizaOP•3mo ago

EbenezerIbizaOP•3mo ago

Is there somewhere I can query the database to figure out what version it thinks it is?

Prastoin•3mo ago

Everything should be within your default database not public one

EbenezerIbizaOP•3mo ago

Prastoin•3mo ago

Or maybe you were talking about postgres user Your workspace version is at default.core.workspace.version

EbenezerIbizaOP•3mo ago

Well, the data is loaded now... Yes, I see it is 0.51.1 And I have redeployed twenty using TAG=0.51

Prastoin•3mo ago

Perfect

EbenezerIbizaOP•3mo ago

So all should be good, right?

Prastoin•3mo ago

In theory yeah

EbenezerIbizaOP•3mo ago

But still cannot login

Prastoin•3mo ago

Please share both the ui and server error

EbenezerIbizaOP•3mo ago

UI error: column User__User_workspaces.defaultAvatarUrl does not exist

Prastoin•3mo ago

also run in your twenty-server container yarn command:prod cache:flush

EbenezerIbizaOP•3mo ago

server error is from the container logs?

Prastoin•3mo ago

Is your server up ?

EbenezerIbizaOP•3mo ago

 $ podman exec -it twenty-db yarn command:prod cache:flush
Error: crun: executable file `yarn` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found

 $ podman exec -it twenty-db yarn command:prod cache:flush
Error: crun: executable file `yarn` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found

Yes, the server is up Ah. It is not the -db pod

Prastoin•3mo ago

Is it healthy ? Are you able to hit its healtz endpoint ?

EbenezerIbizaOP•3mo ago

I dont know about healtz endpoint? What does this mean?

Prastoin•3mo ago

Your server is not healthy

EbenezerIbizaOP•3mo ago

Was I supposed to run "podman exec -it twenty-server yarn command:prod cache:flush" somewhere during my upgrade? or after the db reload?

Prastoin•3mo ago

I'm afraid the dump you have did not contain everything such as enums and extensions 🤔 Does not seem related right now

EbenezerIbizaOP•3mo ago

Not sure if you can see all my messagse. I get these errors from discord bot "Clyde": Your message could not be delivered. This is usually because you don't share a server with the recipient or the recipient is only accepting direct messages from friends. You can see the full list of reasons here: https://support.discord.com/hc/en-us/articles/360060145013 It surprises me that the server is not healthy.... But not sure what to do about it or why it happened. Maybe I just need to restart the containers?

Prastoin•3mo ago

could you please run in your twenty-server

yarn database:migrate:prod
yarn command:prod upgrade

yarn database:migrate:prod
yarn command:prod upgrade

Yes restarting won't harm

EbenezerIbizaOP•3mo ago

migrate:prod gave:

EbenezerIbizaOP•3mo ago

yarn command:prod upgrade

Prastoin•3mo ago

Are you you recovered the correct dump ?

EbenezerIbizaOP•3mo ago

Prastoin•3mo ago

Looks like you've recovered a dump from after the 0.55 migrate

EbenezerIbizaOP•3mo ago

Oh yes, I am sure it is the correct dump. Why am I sure? Because the core.workspace said version=0.51.1

Prastoin•3mo ago

Unrelated You have database versioning ( migrate ) and workspace versioning ( workspace version )

EbenezerIbizaOP•3mo ago

Prastoin•3mo ago

The workspace upgrade is protected by the workspace version in database What's your dump creation date please ?

EbenezerIbizaOP•3mo ago

I no longer have the dump date because I made a copy of the dump to databases_backup.sql.orig before I changed the original dump to not create the databse and role:

EbenezerIbizaOP•3mo ago

databases_backup2.sql is the dump made AFTER the upgrade

Prastoin•3mo ago

Could you please inspect your restored database core schema and list all its tables

EbenezerIbizaOP•3mo ago

Would it help if I delete my containers and the volumes and then recreate for version0.51 and then load the dump? Sure

Prastoin•3mo ago

Indeed this is a good idea, but you will have to hack through your dump file a bit Also we need, in the first place, to assure your dump integrity

EbenezerIbizaOP•3mo ago

Not sure where to put the schema dump? Is it typeorm_migrations we need?

EbenezerIbizaOP•3mo ago

I have restarted the containers... I have not tried deleting/recreating them yet as I am not sure if we expect this to have any effect. Hmm... One thing I just notice when I restart the pods:

Prastoin•3mo ago

What do you mean ? Yes

EbenezerIbizaOP•3mo ago

It says twenty:latest

Prastoin•3mo ago

Could you please inspect your restored database core schema and list all its tables

EbenezerIbizaOP•3mo ago

So maybe it is ignoring my TAG=0.51 in .env?

 $ head -4 .env
#TAG=latest
#TAG=0.55
TAG=0.51

 $ head -4 .env
#TAG=latest
#TAG=0.55
TAG=0.51

Prastoin•3mo ago

echo $APP_VERSION

echo $APP_VERSION

in your twenty-server

EbenezerIbizaOP•3mo ago

$ podman exec -it twenty-server /bin/sh
/app/packages/twenty-server $ echo $APP_VERSION
v0.55.9
/app/packages/twenty-server $

$ podman exec -it twenty-server /bin/sh
/app/packages/twenty-server $ echo $APP_VERSION
v0.55.9
/app/packages/twenty-server $

Huh? So it is ignoring .env? If I do podman-compose down ; podman-compose pull; podman-compose up -d then it should be running the twenty containers corresponding to the version specified in TAG=, right? The upgrade guide says:

We recommend consuming major.minor version such as 0.53

So I guess it is not because I need to specify 0.51.1, right? Is it because I need TAG=v0.51 ? (ie. adding the v?) If this is the case, then it does not correspond to what the upgrade docs are saying.... But I have sometimes seen some apps requiring the v be included in the version spec?

Prastoin•3mo ago

You can find docker hub versioning here https://hub.docker.com/r/twentycrm/twenty/tags Sometimes it does not relate the github tags docker compose down all your containers after changing the tag and docker-compose up them back

EbenezerIbizaOP•3mo ago

The tags seem to include the v Would be good to have the upgrade docs fixed to include this... So now I have changed the tag to TAG=v0.51 Should I just do: podman-compose down ; podman-compose pull; podman-compose up -d ? And then restore the db from backup perhaps? Or no?

Prastoin•3mo ago

To include what ? Yes please restore once we've verified the APP_VERSION in your twenty-server

EbenezerIbizaOP•3mo ago

To include the v in TAG: https://twenty.com/developers/section/self-hosting/upgrade-guide

EbenezerIbizaOP•3mo ago

It should say "We recommend consuming "v$major.$minor" version such as v0.53" or something like that

EbenezerIbizaOP•3mo ago

Well, this is odd:

Prastoin•3mo ago

[+] Running 3/3
 ✘ worker Error               failed to resolve ref...               0.5s 
 ✘ change-vol-ownership Error context...                             0.5s 
 ✘ server Error               context canceled                       0.5s

[+] Running 3/3
 ✘ worker Error               failed to resolve ref...               0.5s 
 ✘ change-vol-ownership Error context...                             0.5s 
 ✘ server Error               context canceled                       0.5s

EbenezerIbizaOP•3mo ago

This is after I did change TAG=v0.51 and then did the down ; pull; up

Prastoin•3mo ago

Idk for podman but providing the wrong tag fails for docker-compose

EbenezerIbizaOP•3mo ago

Do I perhaps need to be more specific, like TAG=0.51.1 ?

Prastoin•3mo ago

Will update the docs on the fly no

EbenezerIbizaOP•3mo ago

Sorry TAG=v0.51.1 ?

Prastoin•3mo ago

Should not have an impact But as your current workspace version is v0.51.1 would still use this version

EbenezerIbizaOP•3mo ago

I found the problem! Or at least: I found out why TAG was not in effect There was an issue in the podman-compose.yml which made it ignore it. I have fixed this now. It is starting up Fails to start:

Trying to pull docker.io/library/redis:v0.51...
Error: initializing source docker://redis:v0.51: reading manifest v0.51 in docker.io/library/redis: manifest unknown: manifest unknown
exit code: 125
podman start twenty-redis
Error: no container with name or ID "twenty-redis" found: no such container
exit code: 125

Trying to pull docker.io/library/redis:v0.51...
Error: initializing source docker://redis:v0.51: reading manifest v0.51 in docker.io/library/redis: manifest unknown: manifest unknown
exit code: 125
podman start twenty-redis
Error: no container with name or ID "twenty-redis" found: no such container
exit code: 125

Prastoin•3mo ago

Congrats ! If could you please share the issue so it could help other podman user, would be lovely !

EbenezerIbizaOP•3mo ago

Yes, I will. But I think the issue is not resolved, though. Just the stuff with it keeping getting the :latest version instead of TAG. But it fails to start and I have no idea why. It gives some error with unknown manifest? The issue was that in podman-compose.yml, it would say image: docker.io/twentycrm/twenty:latest insteaf of image: docker.io/twentycrm/twenty:${TAG} and so on I guess the issue is that I added the TAG value to all images but it does not exist for redis image. So I reverted that to :latest.

EbenezerIbizaOP•3mo ago

Now I am able to get furhter in the app. When I log in, I get to the next step after username. I then enter password and now I get this error:

EbenezerIbizaOP•3mo ago

What would be a good next step from here?

Prastoin•3mo ago

Hello could you please double check the following: - twenty server is on the correct version - share your core schema tables in your just restored database

EbenezerIbizaOP•3mo ago

How do check server version?

Prastoin•3mo ago

echo $APP_VERSION in twenty-server

EbenezerIbizaOP•3mo ago

/app/packages/twenty-server $ echo $APP_VERSION v0.51.14 How do I send the dump? core schema dump

Prastoin•3mo ago

Please do not share your dump Just the list of the tables Of the successfully restored database

EbenezerIbizaOP•3mo ago

What is the easiest way to get just the list of tables from core schema?

Prastoin•3mo ago

Connecting to your twenty-db container and using psql

EbenezerIbizaOP•3mo ago

~/apps/twenty $ podman exec -it twenty-db psql -U twenty
psql (16.9 (Debian 16.9-1.pgdg120+1))
Type "help" for help.

twenty=# \dt
Did not find any relations.
twenty=# \dt core.*
                              List of relations
 Schema |                       Name                        | Type  | Owner
--------+---------------------------------------------------+-------+--------
 core   | _typeorm_generated_columns_and_materialized_views | table | twenty
 core   | _typeorm_migrations                               | table | twenty
 core   | appToken                                          | table | twenty
 core   | approvedAccessDomain                              | table | twenty
 core   | featureFlag                                       | table | twenty
 core   | keyValuePair                                      | table | twenty
 core   | postgresCredentials                               | table | twenty
 core   | twoFactorMethod                                   | table | twenty
 core   | user                                              | table | twenty
 core   | userWorkspace                                     | table | twenty
 core   | workspace                                         | table | twenty
 core   | workspaceSSOIdentityProvider                      | table | twenty
(12 rows)

twenty=#

~/apps/twenty $ podman exec -it twenty-db psql -U twenty
psql (16.9 (Debian 16.9-1.pgdg120+1))
Type "help" for help.

twenty=# \dt
Did not find any relations.
twenty=# \dt core.*
                              List of relations
 Schema |                       Name                        | Type  | Owner
--------+---------------------------------------------------+-------+--------
 core   | _typeorm_generated_columns_and_materialized_views | table | twenty
 core   | _typeorm_migrations                               | table | twenty
 core   | appToken                                          | table | twenty
 core   | approvedAccessDomain                              | table | twenty
 core   | featureFlag                                       | table | twenty
 core   | keyValuePair                                      | table | twenty
 core   | postgresCredentials                               | table | twenty
 core   | twoFactorMethod                                   | table | twenty
 core   | user                                              | table | twenty
 core   | userWorkspace                                     | table | twenty
 core   | workspace                                         | table | twenty
 core   | workspaceSSOIdentityProvider                      | table | twenty
(12 rows)

twenty=#

Like this?

Prastoin•3mo ago

Yep ! if you could please list all the rows

EbenezerIbizaOP•3mo ago

So not just core.*?

Prastoin•3mo ago

Ah sorry my bad looks good ok You should be able to start the upgrade on a good basis !

EbenezerIbizaOP•3mo ago

But shouldnt I be able to login into the app now?

Prastoin•3mo ago

Are you facing this ?

EbenezerIbizaOP•3mo ago

Yes

EbenezerIbizaOP•3mo ago

But first time I get THIS error:

EbenezerIbizaOP•3mo ago

Perhaps it is just repeated error twice.... Should I not be able to log in now? Yes. This is my current state. It is not clear to me whether this is expected and I should proceed with incremental upgrade. Or this is a problem and I should fix this before procedding.

Prastoin•3mo ago

If you workspace version is 0.51.14 you should not have an issue please run a in your twenty-server

yarn database:migrate:prod

yarn database:migrate:prod

Before upgrading incrementally I would recommend having a stable twenty instance Have you cleared the cache also ?

EbenezerIbizaOP•3mo ago

I have run yarn database:migrate:prod. It did not produce errors After this, I am able to log in to the app Does this mean that I am now on a working 0.51 state? And I can proceed with incremental upgrade?

Prastoin•3mo ago

After double checking that your workspace version is equal to your 0.51 twenty instance yes !

EbenezerIbizaOP•3mo ago

How do I check this?

EbenezerIbizaOP•3mo ago

Is it this?

EbenezerIbizaOP•3mo ago

Or is that not a sure check?

EbenezerIbizaOP•3mo ago

Or this?

Prastoin•3mo ago

default.core.workspace.version === workspace version APP_VERSION in twenty-server === Twenty instance version

EbenezerIbizaOP•3mo ago

Not sure what this means? I need to get this from the DB ? default.core.workspace.version

Prastoin•3mo ago

this

EbenezerIbizaOP•3mo ago

I do not have schema "default". But I have this:

EbenezerIbizaOP•3mo ago

So I guess 0.51.1 != 0.51.14 But not sure if that third number is critical?

Prastoin•3mo ago

By definition it should not be critical as upgrade commands should never be added to a patch version You can go for 0.52 please always pick the latest patch version or directly v0.52

EbenezerIbizaOP•3mo ago

OK. So I am good to proceed with inc. upgrade from 0.51 -> 0.52? Correct? Sorry, but this is a bit confusing for me and I am a bit afraid to mess this up again....

Prastoin•3mo ago

Yes you should be good ! No worries, and now you know how to restore to the 0.51 in case I would recommend making a dump at each versions

EbenezerIbizaOP•3mo ago

A database dump? Or also dump other stuff? Thanks for confirming $ podman exec -it twenty-db pg_dumpall -U twenty > databases_backup-v0.51.1-$(date -I).sql

Prastoin•3mo ago

You're welcome

EbenezerIbizaOP•3mo ago

I think it would make sense to update the upgrade guidelines based on what we found here: 1: TAG must include "v" in the version spec. E.g. v0.51 and NOT 0.51 2: The instructions to backup/restore database does not work due to missing handling of db/role creation. There should be instructions on how to manually DROP DATABASE and coment out CREATE DATABASE and CREATE ROLE statements from the dump I think I have successfully upgraded to 0.52 but how do I verify? I have completed running: yarn database:migrate:prod yarn command:prod upgrade No errors. Do I just login to the app?

EbenezerIbizaOP•3mo ago

Is the verification to log in to the app and open admin panel?

Prastoin•3mo ago

Check workspace version and twenty version And following related upgrade guide docs https://twenty.com/developers/section/self-hosting/upgrade-guide

EbenezerIbizaOP•3mo ago

$ echo 'echo $APP_VERSION; exit' | podman exec -i twenty-server /bin/sh v0.52.11 $ echo 'SELECT version FROM core.workspace;' | podman exec -i twenty-db psql -U twenty version --------- 0.52.11 (1 row)

Prastoin•3mo ago

Looks good

Prastoin•3mo ago

Please note that

EbenezerIbizaOP•3mo ago

OK. So since I am using podman, I guess I will have to figure out what was changed in the Dockerfile such that I can update my podman

Prastoin•3mo ago

I'm not sure to understand The dockerFile is used to built the image you will be pulling from the docker hub You don't have anything to do

EbenezerIbizaOP•3mo ago

Ah. I see! Sorry, I was thinking of docker-compose.yml

Prastoin•3mo ago

No worries all good ! But indeed on any docker compose yml file the update has to be done manually ( not the case here )

EbenezerIbizaOP•3mo ago

So basically to get to 0.53, I just have to stop twenty; change .env TAG=v0.53 ; start twenty ; that is it? Then I can repeat for 0.54; and so on? So I wanted to do this. And for that I wanted to do the db backup. But it fails: No. The db backup is not what fails. Sorry. The getting version from twenty instance is. This is what happens when I need to get the version by getting a shell on the instance:

$ podman exec -i twenty-server /bin/sh
Error: can only create exec sessions on running containers: container state improper

$ podman exec -i twenty-server /bin/sh
Error: can only create exec sessions on running containers: container state improper

´ If I try to open the app in a browser, I get 502 Bad Gateway So perhaps the upgrade to 0.53 did not go well afterall? Restarted containers and now it works... strange.... trying upgrade again.... I think it succeeded:

~/apps/twenty $ echo 'echo $APP_VERSION; exit' | podman exec -i twenty-server /bin/sh
v0.54.6
~/apps/twenty $ echo 'SELECT version FROM core.workspace;' | podman exec -i twenty-db psql -U twenty
 version
---------
 0.54.6
(1 row)

~/apps/twenty $ echo 'echo $APP_VERSION; exit' | podman exec -i twenty-server /bin/sh
v0.54.6
~/apps/twenty $ echo 'SELECT version FROM core.workspace;' | podman exec -i twenty-db psql -U twenty
 version
---------
 0.54.6
(1 row)

~/apps/twenty $ echo 'SELECT version FROM core.workspace;' | podman exec -i twenty-db psql -U twenty version --------- 0.55.9 (1 row) ~/apps/twenty $ echo 'echo $APP_VERSION; exit' | podman exec -i twenty-server /bin/sh v0.55.9 YAY! Thank you @prastoin for your help!

Prastoin•3mo ago

Congrats @EbenezerIbiza ! You're welcome !

Gaming

Programming

Upgrade failed - is my deployment broken now? Can I fix it?

Did you find this page helpful?