Weird micro-stutters when moving from chunk to chunk, only on server

Hi, I'm hosting a modded server, and I'm getting some micro-stutters when moving from chunk to chunk. The weird part is that they disappear after going into a single player world, and then going back in the server. I'm pretty sure that Distant Horizons is the cause of the stutters since they disappear if the mod isn't installed.

Before going into a singleplayer world: