Unable to start apps after Update to 4.0
Hey guys,
updated today but i'm getting problems with the networks.
Roughly half of my apps wont start.
58 Replies
How many apps do you have?
Please share the result of ./runtipi-cli debug
Damn did docker run out of subnets?
Looks like it went out of addresses in the network
Sorry for the delay - work stuff π
Maybe you have something special in your
user-config/tipi-compose.yml?
Otherwise I suggest to remove some containers manually
Check with docker pa
docker psYou can increase the addresses by editing the docker daemon config I think
I have a grand total of 9 applications that can start. This seems to be the max as I can only install or start another, if I completely uninstall one of the 9. All I have in my tipi compose is traefik stuff. Dashboard port and DNS token.
Try to
docker network prune maybe it will release some addresses
Have you done custom things in your docker install /etc/docker/daemon.json? To limit address pool or something like using a different subnet?No but i have noticed something. Several apps are in the network 172.x.0.0 but there are others in 192.168.x.0. the x in the 172 network are going up 1 at a time but in the 192 network there are going up in the 3rd octave in multiples of 16 so it runs out of addresses very quickly
I've also removed all apps 1 by 1 and began reinstalling them.
According to Portainer i have 33 networks which seems to be 30 apps and the usual 3 you get on the host
Also to answer your questions - Pruning hasnt helped and i havent stepped outside the default directory in opt
I've found the root cause of your issue, since we don't define the subnets explicitly Docker is using its default scheme. After 15 bridge networks docker runs out of available addresses
ah-ha! brilliant π
I'll release a patch fix later today
If you are in a hurry we can add custom configs to your apps to use specified networks
No hurry at all! i took some stuff offline and have backups so can fix it later without issue! thank you for looking into it
Change is not as trivial as expected! Will continue tomorrow
I have the fix ready, I'm releasing a beta version. Are you willing to test?
absolutely
Cool, wait a couple of minutes I'm going to update my instance with this beta to make sure it works first
ok cool
Count me in
Okay so it works on my side. Try
sudo ./runtipi-cli update v4.0.3-beta.1Now each app has a dedicated internal subnet
What the heck
Which will cap the number of apps to 244 instead of 30 or so
What does this do
Ipam
Docker default ip scheme uses a narrower pool
We can increase this further
And after 15 networks you start to have conflicts by default
Right now the subnet mask is like 255.255.255.0
We can increase later yes. For now I think it is enough
Like just do /16
255.255.0.0
65534 addresses
Should be enough no?
It's not about the number of addresses in the network
Wait what
Installed and apps coming up now - will let you know if i get the error shortly π
It's about the pool running out of addresses
Nah fk this I am confused now
Remind me to never touch networking stuff
No error - system seems a lot more agile - Installs either with being exposed or not are all fine when installing. Thank you - perhaps you guys should release this solution now and take a few days to think it over. Perhaps something more elegant will come to you when youre not under pressure. I wouldnt give up either. Containered networking by default is a major plus point for Tipi
Thank you again
Excellent! Happy that we have been able to fix this issue
Sorry guys seemed to have hit another networking problem and cant install more apps -
Is the issue only with onedev?
no 3 apps that i have tried since
Sorry the authentik one is different
Can you run
docker network inspect $(docker network ls -q) | jq -r '.[] | "\(.Name): \(.IPAM.Config[0].Subnet // "N/A")"' (install jq if needed)Could you with this one?
It prints a pretty table
Discord saying too many characters so going into a file each time
Okay everything looks fine really, so probably you have a subnet that didn't get correctly registered in the database. Can you try to do a
./runtipi-cli restart and see if this solves the issue? If not, I'll add an extra check to see if the network is free on the system itselfsure
Same again im afraid -
Okay sorry about this bug, I'll give you another update to fix this. Can you just share me the
apps/migrated/<app>/docker-compose.yml file of one of the apps that fails to install?
It should have created this file before failinghmmm keeps deleting the file
This is extremly odd, the network is not in use π€
I'll dive in after work
Are you willing to try
v4.1.0-alpha.1?
It includes a potential fix for itHey Nicolas sorry i havent replied - been busy! Would you like me to try it out now?
Yes sure when you want
Since I'm not really sure why you have this issue, this version includes a more robust check directly against docker api to look for free networks
Not 100% sure it'll work, if not I'll implement a retry strategy with a random subnet each time
Installed and apps are installing again
any data youneed me to get for you?
Okay that's good to know
I think I have all I need, I'll still implement the more robust retry mechanism just in case
Thank you
NP! thank you for your efforts!
Just as a note, youβll see other alpha updates even beta. I suggest you keep the one you have until we have 4.1 stable out
Hi, I've just installed the final version of 4.1 and I have the same issue. Multiple apps don't want to start π
Same error? Could you share your logs?
Iβve dug in the discord and found I had to close all apps with the command sudo docker stopβ¦
It fixed my issue but I have no idea how this happened. Thank anyway π