ERROR: could not resize shared memory segment "/PostgreSQL.1432368484" to 12615680 bytes:

Hi, I am using Postgres database in Railway and I am getting this error. Can someone please help me figure out why this is happening? All of the solutions I see online tells me to change shm_size of the database. But I don't think there is a way to change shm_size in Railway. This is only happening for some of the complex or large queries that returns a lot of rows. How to fix this? What can I do to solve this? Project ID: ef4b4933-5e06-4f9f-93d1-1153cb313e98 Some of the errors I am seeing are these: ERROR: could not resize shared memory segment "/PostgreSQL.1432368484" to 12615680 bytes: No space left on device FATAL: terminating background worker "parallel worker" due to administrator command ERROR: could not resize shared memory segment "/PostgreSQL.1662690888" to 12615680 bytes: No space left on device background worker "parallel worker" (PID 35856) exited with exit code 1 My usage dashboard says Your plan includes up to 8 GB of RAM, 8 vCPU, and 100 GB of Shared Disk. Manage your Plan. and I am no where near those limits.
Solution:
the default shm size on railway is 64mb and there is no way to increase this
Jump to solution
65 Replies
Percy
Percy7mo ago
Project ID: ef4b4933-5e06-4f9f-93d1-1153cb313e98
Solution
Brody
Brody7mo ago
the default shm size on railway is 64mb and there is no way to increase this
Bhanu Teja P
Bhanu Teja P7mo ago
@Brody Yes, I know. Hence this question. Will wait for someone from team to suggest a solution to this.
Brody
Brody7mo ago
ive talked to the team about this myself, they don’t seem too interested in this issue, you may need to host the database elsewhere
Bhanu Teja P
Bhanu Teja P7mo ago
Wow! Didn't expect that. Ok, then I guess I need to move my database off of Railway!
Brody
Brody7mo ago
yeah for your workload you would need to, as this is definitely a blocker they don’t seem too interested in solving unfortunately
Bhanu Teja P
Bhanu Teja P7mo ago
Got it. Thanks Brody!
Brody
Brody7mo ago
no problem, sorry for the letdown here
Bhanu Teja P
Bhanu Teja P7mo ago
Are these any team members who checks the conversations in discord? or is it all community managed?
Brody
Brody7mo ago
This is a community forum, you would need to be on Pro in order to get support from the team but having Pro wouldn't help here as other Pro users have been left in the dark on this issue as well
Pepijn
Pepijn7mo ago
Not really sure if this answers is satisfying, but I will ask the team directly as well on Slack. Should be as simple as passing a variable to the docker run command.
Brody
Brody7mo ago
Yes in theory it's that simple, but they very likely don't have a mechanism in place that would allow them to tack on an extra flag to a specific users container run command the team has directly told me and another pro user they can't offer custom shm sizes, so unless you are actually enterprise then you may get the same answer I did: you would need to tune postgres to not use as much shared memory
Melissa
Melissa7mo ago
hey all! Brody is correct, you can't pass a flag to your start command, and we hadn't been able to prioritize a fix for this until recently, but I'm happy to report that we do have an environment variable now that enables control over shm_size. the variable name is RAILWAY_SHM_SIZE_BYTES which takes a value of bytes (as the name implies). let us know if you have any problems
Brody
Brody7mo ago
I shall test that with my database! edit, works perfectly!
christian
christian7mo ago
@Bhanu Teja P, flagging Melissa's note above which should unblock you by allowing you to control shm_size using the RAILWAY_SHM_SIZE_BYTES variable.
Bhanu Teja P
Bhanu Teja P7mo ago
@Christian where do I add this env variable? In my repo which connects to the database? Or should I add this in the env variables of the database? @Brody where did you add this? And how much did you set it to?
Brody
Brody7mo ago
please do not tag the team, you add it to your service variables
Bhanu Teja P
Bhanu Teja P7mo ago
Hmm… ok. Can you please share where that is? Service variables of the database?
Brody
Brody7mo ago
correct
christian
christian7mo ago
You can follow these steps: 1. Click on your database on the canvas 2. In the service settings pane, click Variables from the top menu 3. Click New Variable 4. Enter VARIABLE_NAME = shm_size // VALUE = your custom shm size 5. Click Add
Bhanu Teja P
Bhanu Teja P7mo ago
Thanks Christian. Is there a way to directly reach out to your team when needing customer support instead of posting it on a community managed forum. Previously, it used to be so simple to contact the team via discord. Now it looks like my ticket automatically got marked as resolved without the team even seeing it
Brody
Brody7mo ago
^
Bhanu Teja P
Bhanu Teja P7mo ago
Yeah, but you said don't take pro as they would say the same thing. If I would have listened to that, I would have migrated to another service
Brody
Brody7mo ago
I can only provide the information I am aware of, I did not know they were going to add this variable the statement is true regardless
Bhanu Teja P
Bhanu Teja P7mo ago
Hmm... ok, thanks. I set the value to 512000000 (assuming this means 512 MB) Hoping this will solve the problem
Brody
Brody7mo ago
please don't set it anywhere near that high, as a hobby user that would likely be far above the limit for shm sizes on the hobby plan when they implement the limits set it to 128MB, or 134217728 in bytes
Bhanu Teja P
Bhanu Teja P7mo ago
I tried 128 MB, but it's still not enough for my usecase
Brody
Brody7mo ago
the error above states you only went 12mb out of memory
Bhanu Teja P
Bhanu Teja P7mo ago
I have no idea. I tried 128 MB, but it still throws the same error. I set it to 512 MB, and no error let me set it back to 128 MB again and retry it
Brody
Brody7mo ago
we may set the default to 160mb so try that
Bhanu Teja P
Bhanu Teja P7mo ago
160 MB didn't work Trying 256 MB next
Brody
Brody7mo ago
please provide the errors
Bhanu Teja P
Bhanu Teja P7mo ago
Same error
Raw query failed. Code: `53100`. Message: `db error: ERROR: could not resize shared memory segment "/PostgreSQL.1645299550" to 12615680 bytes: No space left on device`
Raw query failed. Code: `53100`. Message: `db error: ERROR: could not resize shared memory segment "/PostgreSQL.1645299550" to 12615680 bytes: No space left on device`
Brody
Brody7mo ago
yeah same error, but the number would change
Bhanu Teja P
Bhanu Teja P7mo ago
It didn't change My guess is there are multiple of these queries running in parallel, and together combined is more than 160 MB otherwise, it would not have worked for 512 MB too
Brody
Brody7mo ago
this is starting to sound not like a hobby usecase if I'm being honest
Bhanu Teja P
Bhanu Teja P7mo ago
what's a hobby usecase? Setting it to 256 MB worked. I am going to leave it at this.
Brody
Brody7mo ago
that's good to hear
Bhanu Teja P
Bhanu Teja P7mo ago
Thanks for all the help @Brody
Brody
Brody7mo ago
no problem
Bhanu Teja P
Bhanu Teja P7mo ago
Now, I don't have to write a migration script anymore 😅
Brody
Brody7mo ago
I'll preemptively say this, when / if you use your 5gb volume up on the database, you will need to upgrade to pro
Bhanu Teja P
Bhanu Teja P7mo ago
Where do I check this?
Brody
Brody7mo ago
click on the volume attached to postgres not even close to where you should be looking haha
Bhanu Teja P
Bhanu Teja P7mo ago
This?
No description
Brody
Brody7mo ago
yep, looks like you are soon going to hit the limit soon, once that happens your database becomes inaccessible
Bhanu Teja P
Bhanu Teja P7mo ago
In Pro, do I get a higher size? How much higher?
Brody
Brody7mo ago
pro users get 50gb
Bhanu Teja P
Bhanu Teja P7mo ago
Nice! Will upgrade to Pro
Brody
Brody7mo ago
once on pro don't feel bad if you need to set your shm size higher
Bhanu Teja P
Bhanu Teja P7mo ago
As soon as I upgraded to Pro, I see this
No description
Brody
Brody7mo ago
have you moved the project over to the pro workspace?
Bhanu Teja P
Bhanu Teja P7mo ago
yeah what do I do?
Brody
Brody7mo ago
please confirm that the project is now located in the pro workspace
Bhanu Teja P
Bhanu Teja P7mo ago
It is
No description
Bhanu Teja P
Bhanu Teja P7mo ago
No description
Brody
Brody7mo ago
show me what's in the drop-down for the workspace selector on the left of this screenshot
Bhanu Teja P
Bhanu Teja P7mo ago
No description
Brody
Brody7mo ago
interesting, can you quickly check your transactions for the card you used to upgrade, where you charged $20?
Bhanu Teja P
Bhanu Teja P7mo ago
Checking
Bhanu Teja P
Bhanu Teja P7mo ago
There was an unpaid invoice. Paid it. Now, it's back to normal.
No description
Bhanu Teja P
Bhanu Teja P7mo ago
Any idea why this would still show 5 GB?
No description
Brody
Brody7mo ago
you would need to grow it, it's not done automatically as growing the volume means downtime, so it's left to the users discretion, you can grow it when you want to in it's settings menu
Bhanu Teja P
Bhanu Teja P7mo ago
Understood. Again, thanks for all the help!
Brody
Brody7mo ago
no problem
Want results from more Discord servers?
Add your server