Impact of ID Pool Initialisation on Query Performance
Greetings everyone,
We're currently operating a JanusGraph setup with
cluster.max-partitions set at 1024 and ids.num-partition at 10. Our primary goal is to ensure high availability for the cluster instances. However, we've noticed that the initialization of the ID pool is causing disruptions, during server restarts. The root cause seems to be the initialisation of ID pool threads for each partition until every partition has a ID pool.
Upon server restart, the ID pool is initialised based on write operations. Unfortunately, this process has been negatively impacting the performance of query execution. To mitigate this challenge, we're exploring the possibility of implementing an eager initialisation approach for the ID pool threads. Is there a way to achieve it?
Thank you for your attention and assistance in addressing this matter.2 Replies
Hi Everyone, we are still facing the above issue, any help will be greatly appreciated. TIA.
@Boxuan Li any suggestion on this, currently the idpool gets initialised only after the instance start receiving write requests and the queries start to timeout for ~15 minutes until all the idpool are initialised.
Is there a way to initialise the idpools before we send the write throughput to the janusgraph instance?
You might need to dive into the source code. I don't have an answer to that, unfortunately. You mentioned "we're exploring the possibility of implementing an eager initialisation approach for the ID pool threads". Does that work out?