Typical dev -> main branch workflow

what's the recommended workflow when working with a simple project that has the main and dev branches. I do changes to the dev branch, add data that works within the new schema and once everything is tested and ready, I want to move the new schema WITH the data to the main branch. What's the best/easiest way of doing this?

8 Replies

quickest-silver•8mo ago

You will need to reapply the changes you made to the dev branch to your main branch If you generated the data using SQL you can re-run the same query against the main branch What's your tech stack? Are you using an ORM?

harsh-harlequinOP•8mo ago

I'm working with next, payloadcms (which uses drizzle behind the scenes). What I want is a solution that could simply just replace the production branch by the data/schema from the dev branch. We used to work with mongodb on our projects, and that was as easy as two commands which instantly clean the production db and replace its schema/data with the dev db. When I started using neon, I was super excited by the branches, because I thought that I could actually use those just like I do with code (git), whereas merging schema/data is super easy and straightforward. Or at least be able to make the dev branch the default and root one going forward. But after making the dev branch the default, it doesn't let me remove the main branch as its the root branch. How I see a good workflow here is: 1. Branch out from main to dev. 2. Make changes to schema, add data to dev branch. 3. Test and make sure that dev branch works well with the website I'm working on in dev/staging. 4. Replace main branch with dev branch schema and data without fuss. At this point I've spent two days searching for a good solution. Wrote my own scripts that use pg_dump and pg_restore to dump the dev branch data/schema and restore onto main branch, that was cleaned before restoring. But I'm constantly getting a bunch of errors. In your docs you mention that those role/privilege errors can be fixed using -O --no-owner args, but that does absolutely nothing and I failed to find enough info on how those problems are handled if I use those args (does it mean that I could loose data? loose functionality sometime in the future because of that workflow?)

pg_restore: while PROCESSING TOC:
pg_restore: from TOC entry 2081; 826 16388 DEFAULT ACL DEFAULT PRIVILEGES FOR SEQUENCES cloud_admin
pg_restore: error: could not execute query: ERROR:  must be member of role "cloud_admin"
Command was: ALTER DEFAULT PRIVILEGES FOR ROLE cloud_admin IN SCHEMA public GRANT ALL ON SEQUENCES TO neon_superuser WITH GRANT OPTION;


pg_restore: creating DEFAULT ACL "public.DEFAULT PRIVILEGES FOR TABLES"
pg_restore: from TOC entry 2080; 826 16387 DEFAULT ACL DEFAULT PRIVILEGES FOR TABLES cloud_admin
pg_restore: error: could not execute query: ERROR:  must be member of role "cloud_admin"
Command was: ALTER DEFAULT PRIVILEGES FOR ROLE cloud_admin IN SCHEMA public GRANT SELECT,INSERT,REFERENCES,DELETE,TRIGGER,TRUNCATE,UPDATE ON TABLES TO neon_superuser WITH GRANT OPTION;

pg_restore: while PROCESSING TOC:
pg_restore: from TOC entry 2081; 826 16388 DEFAULT ACL DEFAULT PRIVILEGES FOR SEQUENCES cloud_admin
pg_restore: error: could not execute query: ERROR:  must be member of role "cloud_admin"
Command was: ALTER DEFAULT PRIVILEGES FOR ROLE cloud_admin IN SCHEMA public GRANT ALL ON SEQUENCES TO neon_superuser WITH GRANT OPTION;


pg_restore: creating DEFAULT ACL "public.DEFAULT PRIVILEGES FOR TABLES"
pg_restore: from TOC entry 2080; 826 16387 DEFAULT ACL DEFAULT PRIVILEGES FOR TABLES cloud_admin
pg_restore: error: could not execute query: ERROR:  must be member of role "cloud_admin"
Command was: ALTER DEFAULT PRIVILEGES FOR ROLE cloud_admin IN SCHEMA public GRANT SELECT,INSERT,REFERENCES,DELETE,TRIGGER,TRUNCATE,UPDATE ON TABLES TO neon_superuser WITH GRANT OPTION;

Here's an excerpt I keep getting even when using the docs recommended args for pg_restore ⬆︎ All generated data is done through payloadcms admin panel, and all the data can be generated through a month while the dev/staging project is ongoing with lots of editors working on the content that's stored on the dev branch db In my search for a solution I've actually found a LOT of people asking for the same thing and nobody has an actually good solution. it should kind of work like vercel, where you can "promote" a branch to production and completely replace the previous production deployment with the dev one kind of like promoting dev branch to main as simple as possible without this, I can't seem to understand the point of branches (which you cannot even merge). Spinning up a separate dev db is almost as quick as creating a new dev branch, so why should the branch functionality (albeit very limited) be useful other than this usecase? my dump command looks like this:

pg_dump ${POSTGRES_URL_DEV} -F c -b -v -f ${backupFile}

pg_dump ${POSTGRES_URL_DEV} -F c -b -v -f ${backupFile}

my restore command looks like:

pg_restore -v -O --no-owner -d "${POSTGRES_URL_PRODUCTION}" -c --if-exists "${backupFile}"

pg_restore -v -O --no-owner -d "${POSTGRES_URL_PRODUCTION}" -c --if-exists "${backupFile}"

If this helps understand my issue/solution.

quickest-silver•8mo ago

1. Branch out from main to dev. 2. Make changes to schema, add data to dev branch. 3. Test and make sure that dev branch works well with the website I'm working on in dev/staging. 4. Replace main branch with dev branch schema and data without fuss.

How about this workflow: 1. Branch out from main to dev. 2. Make changes to the schema and make sure that everything works in the preview environment 3. Merge the schema changes and re-apply the schema changes in CI or in your project's build step 4. The data is added to the main branch from the admin panel

I can't seem to understand the point of branches (which you cannot even merge). Spinning up a separate dev db is almost as quick as creating a new dev branch

Branches allow you to spin up a dev database that contains both the schema and data. You won't need to maintain a seed script and you can work with realistic data Merging branches is something we're thinking about. It's a bit tricky because a database that's running in production will typically have dynamic data that's changing (inserts, updates, deletes)

be able to make the dev branch the default and root one going forward

This is possible but it's not exposed in the UI/API yet pg_dump & restore should work and will potentially allow you to have the desired workflow. I'll try to reproduce the error you're running into

harsh-harlequinOP•8mo ago

Adding data by hand from the admin panel would mean downtime while all editors add the same data that was added to the dev branch, which could take days, so that's not a solution, unfortunately. Also, I don't see this as a solution to a problem which could be simply solved by replacing the whole db from the dev branch. You have an awesome feature where you can reset a branch from parent. If this would be possible the other way - meaning you can reset main branch to the dev branch data/schema - this would be extremely easy and useful. Thanks for looking into this for me!

adverse-sapphire•8mo ago

You could use the Restore feature to do what I think you're asking (and the restore API). If you want to swap out the data/schema from production main with the data/schema from your dev/staging you could use the restore feature. Restore keeps the same end point (DATABASE_URL) you have for main but replaces the branch underneath it. So your production DBURL would start out pointing to main and then after the restore of dev/staging -> main you're main branch would be the same as your staging. You'd be left with a `main$date

 branch that has the data difference between the two in case you want to revert.
You could try this out by using

dev/staging and restoring dev/testing` into it so you see how it works.

harsh-harlequinOP•8mo ago

Ok, this is a almost perfect, with a few caveats: 1. Firstly, I think this feature should be surfaced in the console a lot higher, because without you telling me, I would probably never would have even seen it or the fact that you're able to do that. 2. If I restore main to dev, then I have two branches that I cannot remove, even if they're completely redundant now: dev, as it is the parent of the new main, and the main_old branch, because it is the root branch. My proposed solution: 1. Have a quick restore button in the branch dropdown (regardless if the branch is the root one or not), and let the user select which branch to restore from. 2. If it's the root branch being restored, show a checkbox: "Set the new main branch as the root (protected) branch" or something along those lines. This way, the main_old branch will be removable and the new main branch would not have a parent (as even now the fact that it has a parent, doesn't actually have any benefits as you're not able to "reset from parent" If this would be implemented in some way, it would solve all my issues for a pretty standard website project for any agency that's creating marketing websites for clients. Have you also had the chance to look into why it's not possible to dump and restore using pg_dump and pg_restore without errors? I've failed to find a properly working dump and restore command that would not throw errors or warnings, which then cause anxiety and fear that some data might be lost or functionality could be broken after such a restoration.

ambitious-aqua•7mo ago

I'm 100% with OP, using Next.js and Payload on Vercel with Neon. It was super exciting to make a dev branch to get up and running in Preview, but when merging my dev into main, I was bummed to find out I could not. So, +1 for that feature. That said, I was able to pg_dump / pg_restore from dev to main using -x/--no-acl/--no-privileges to work around the ERROR: must be member of role "cloud_admin". Here's my working command set:

pg_dump -Fc -v -d $SRC_CONN -f neon.dump
pg_restore -v -d $DST_CONN --clean --no-privileges neon.dump

pg_dump -Fc -v -d $SRC_CONN -f neon.dump
pg_restore -v -d $DST_CONN --clean --no-privileges neon.dump

I did initially try a restore, but not only did it not work (I don't recall the specifics) but I was not confident the resulting "branch" would be the new root (i.e. branch with no parent), which was my goal.

harsh-harlequin•7mo ago

Can you run these commands instead?

pg_dump -Fc -v -d "$SRC_CONN" -f neon.dump

pg_restore -v -d "$DST_CONN" neon.dump

pg_dump -Fc -v -d "$SRC_CONN" -f neon.dump

pg_restore -v -d "$DST_CONN" neon.dump

Also, can you share where you got those commands from?

Gaming

Programming

Typical dev -> main branch workflow

Did you find this page helpful?