Server crashing when flying (usually over new chunks)
This is partially answered in the FAQ but I just want full clarity on the entire situation and also perhaps any recommendations.
When flying (90% of the time over new chunks) the server often lags heavily behind, and then crashes. Most likely just due to the server not being able to handle it, fair enough. Whilst LODs are present and are already loaded (due to the mod being on the server) they often fade out when trying to land and the server has to catch up and generate the actual real chunks. I ran the spark mod and the culprit was mainly DH so I removed that and the server runs fine now when flying around, still a little slow sometimes to load chunks but doesn't drop below 19 TPS.
The confusing thing is the LODs are already loaded and there, for a decent size around our main play area even for areas we've never been (according to the loaded chunks show on BlueMap), so I would assume the server just needs to load the actual chunks and DH can rest easy since they already exist. Maybe its regrabbing them in case of anything new like buildings etc. Regardless, removing the mod from the server fixed it and clients still see as far as the eye can see with all the LODs previously synced.
My main question, does the DH pregenerator only generate LODs? as I dont want to pregen actual chunks causing them to show up on BlueMap (we like seeing where we've explored or not). And if it does only gen LODs can I pregen a bunch when no one is on, have the mod present when everyone jumps on and let them all sync up, and then remove the mod? I understand future LODs far far away outside the generated area will not generate until we explore them, i can always gen more later using this same process, but thats not going to be for a while. I also understand any structure changes like someone building wont show up until I load them with the mod removed, its just me and a friend and we mostly stick together.
Secondary question, is the potential issue because DH is trying to send updated LODs?
Solution:Jump to solution
You can generate the area you want on the server, then offer the players a download for the DH database file
74 Replies
Default settings will generate LODs only, not chunks. For the best performance when flying around, pregen both chunks and LODs.
You can define a border that stops DH from generating further than some set limit
Ok good to know, is there a way to sync the servers LODs (beyond a clients DH distant) to the clients to then remove the mod?
Only issue with pregen chunks is it will show up in BlueMap which we are avoiding. Flying into new chunks without the mod is no issue, but even flying in generated chunks with DH sometimes caused server to die
Solution
You can generate the area you want on the server, then offer the players a download for the DH database file
Oh thats not a bad idea
Just drag and replace the current DH database file for the server on the client end?
Yup
Ok that could work 🙂 thanks
Assuming your server is using the mod and not the plugin. The two have different database formats
Mod 🙂 fabric
👍
Ok, ill consider this solved, but do you know if DH still tries to send/regenerate new LODs even if they are already present? I assume so to update any built structures. Cause thats where I think the server struggles aha
DH will only regenerate an existing LOD if changes are detected in the region files
is this a set limit from an origin point or around the player? like it wont send more than X radius no matter what the clients radius is set to
You set a center and a radius, just like you would with a worldborder
Hmmm interesting. Not sure what the issue is then. Since removing the mod on the server stopped any issues. Cause it would mostly happen in generated LODs but ungenerated chunks, so LODs shouldnt have any updates since they are untouched
It sounds to me like your server has enough capacity to generate either chunks or LODs individually, but noth both at the same time. Exactly which position the player is in during the crash is probably not as relevant as the size of the ungenerated area around them
Yeah thats what I was thinking too. I could try lowering the render distance serverside but if your flying high speed that probably doesnt matter ahaha
You can also lower DH's request concurrency limit, which will prevent players from requesting as many LODs in parallel
But the LODs already being present doesnt make sense, maybe its trying to generate them far far away, since those probably arent
Might give that a shot, might be good middle ground
Thank you for all your help, super appreciate it!
No worries :) There are a lot of settings in DH you can adjust to tune the performance
I recommend reading through the config file, as there are a lot of comments in there that explain what everything does
Yeah looking now, superb config, others should be just like this aha
There's even a CPU preset setting that lowers or increases the number of threads used
Haha
I guess it makes more sense to lower the CPU preset before adjusting the request concurrency limit
now that I think about it
So these two yeah?
20/sec is probably a lot, 50 is probably fine cause thats just for already generated ones
Correct
Alrighty, ill fiddle around with these and see if i can find a sweet spot
SO minecraft is single threaded right, if my server is running 4 cores (through proxmox, with unlimited use in pterodactyl) 3 threads is safe?
That's a bit of a complex question, and will depend on your server software and other mods too. You can often still see performance gains when increasing the thread count past the core count, but it's all very individual
ok ok noted, another one to fiddle with
But I get what you're aiming for: Always leave at least 1 core free for the main thread, right?
Thats the plan aha
but yeh this isnt a heavily modded server, but I think there is some mods that also like running on a seperate thread
threadRunTimeRatio = "1.0" this is the percentage one your were talking about yeah?I am not too familiar with that one. I mainly work on the plugin 🙈
All good, only option that sounds like it would be
Where did I talk about a percentage setting?
nevermind im trippin, you did mention threads
Ah, so that sounds like you can command it to only use say 70% of each core
Either way ill let you free now aha, ill play around with the settings
Whats the worst that could happen
:KEKW:
Feel free to ping me again if you have more questions. I don't know everything, but might be able to point you in the right direction
or involve someone else who knows
Legends
I'll be honest, I was worried that the whole process of forcing the user to read the FAQ to validate would mean staff would be uptight douches who make you feel stupid for not knowing inner workings of a complicated mod, but you proved me wrong! A+ support 😛
Hahah :D
Thanks!
We got a lot of the same questions over and over, which is why we have that whole process set up
Oh dont get me wrong, its a great system aha. Reminds me of those big companies who refuse to put you through to an agent without first reading relevant documents
Haha, as a user though; those are horrible
But I get it
Also, setting request/s to 10 (instead of 20) and changing the RunTimeRatio to 0.8 (instead of 1.0) seems to have fixed the crashing (at least with just me on)
Will do more rigorous testing but seems like ill be able to nail in a sweet spot
That coupled with pregen a massive LOD radius now, should mostly fix the issues, without needing to remove this mod 🙂
Hey @Jckf (GMT+2) sorry for the ping, but you have context for this (assuming you remember this ahaha)
Im still having issues with lag
Now I tried getting some help in the Spark discord cause i thought I fixed the issue by lowering some settings, thought maybe it was some other mod
But after some deducing and hair pulling, disabling DH on the server pretty much negated any lag I was having
Ok, so whats new? well using the spark profiler I was able to see that it seems DH was do some database requests or something which was causing the lag
spark
spark is a performance profiler for Minecraft clients, servers, and proxies.
Now as stated before, I pregenerated i think it was 550 chunks radius around the base (where I was flying and got this profile)
My client distance is 256 and none of the chunks should be changed so DH on the server should be able to just sit back and relax, but its not?
Any ideas?
This crash report might help
This might just be a case of, server aint big enough for the both of us lol. And I will have to remove DH, just sucks cause any LODs not synced now will have to be loaded by flying over them etc, and LOD updates wont happen. 1st world problems amirite
whoopsie
The Spark trace you are showing shows that 13% of a single thread is being taken up by "DB.prepare()". This is a method that prepares a SQL query for execution against the database. Re-preparing the same queries over and over is inoptimal, but since the code is running on a separate thread it should not cause any lag.
basically I just said DH try access my HDD, my HDD slow, probably the issue. Tinkered with settings to lower them even more, maybe fixed
Guy on spark reckons DH is trying to do SQLite things on the main thread, but I cant read these things anyway lol
And yeah, that crash log was caused by a watchdog timeout, which means it didn't actually crash - but took too long to complete some task
Yeah, usually around 20sec
Either way I think this issue is beyond just optimising, unless these new lower settings do work lol, will see. Probably need a faster disk if its a database thing
Ah, I didn't recognize this as happening on the main thread; but that might be. If so it's a case for @пшш or @BackSun
I turned the settings from 2 cores to 1, idk if thats stopping it from running on the main thread lol.
It seems slightly more stable but I said that last time lol
Whatever is happening on the main thread is probably still happening there, but you might have slowed it down enough to not notice it
Yeah thats my hope, have to give it a day or two to know for sure lol
Disabling DH on the server caused everything to run really smooth, but alas I want to try and avoid that aha
I remember there being some sql queries happening on the server thread that probably shouldn’t be, but that’s probably a better question for pshsh.
I’m more familiar with the single player code.
Welp until the right person is able to take a look aha, ill see how these settings go and report back in a day or two, im off to bed 😛
getTimestampForPos() it seemsHope those profiles help 🙂
Pshsh is probably asleep rn, but is usually up pretty early. I'll bug him again tomorrow if we don't see a response ;p
Cool cool, no rush
spark
spark is a performance profiler for Minecraft clients, servers, and proxies.
Also here is profile for new DH config (1 core, 1.0 ratio, before it was 2 cores, 0.85 or something)
The first 5-10 minutes I was flying around (where the lines go wonky lol)
Doesnt seem to be having the same SQL task choking the server, just regular minecraft stuff
Depending on your server software, you could also try the DH plugin
Where can i find that? That wouldnt work on a Fabric server tho would it?
No, you would have to run Arclight or something similar
There are links in #links-n-downloads though, at the bottom
Arclight and Banner are the two Fabric+Bukkit softwares I know of
Hmm, might look into it. Ill give this some time first
Yeah i completely forgot about that thing
should be trivial to fix, will do when i have time
I think database should be guarded to run only on its own threads
(these were not very stable with DH and other mods/plugins when I tested it for the DH SMP, so I don´t recommend using it)
Noted, I'll patiently wait for the database thread fix then 🙂
check the next build when it finishes
:bingodance: :bingodance: :bingodance: :bingodance:
Where should I look? On modrinth or the git page directly
Nightly builds in #links-n-downloads
Sweet