Very long pause times when caching is enabled

Recently an issue with downloads of a game client that requests every single file while downloading became very slow when the CF cache is turned on. I can see some kind of pattern between those delays when a new batch of files gets downloaded. It downloads a few files, then waits like 15 seconds and does a few new files. Then the same gain. Any ideas on that? I did not have that issue before and it suddenly happened. Enabling dev mode, aka disabling caching is currently a workaround I use.