Splitting Backing ElasticSearch Index To Increase Primary Shards As JG Mixed Index Grows
Has anyone had to resize a ElasticSearch index that's backing a JanusGraph Mixed Index?
Configuration wise it seems you're only able to convey to JanusGraph a singular primary shard & replica count when it creates an ES Index.
I'm projecting to eventually have a couple Mixed Indices exceed what will be reasonable for a single primary shard in the backing ES Index (given the rule of thumb of 10-50GB or 200M documents). So as a configuration default it makes sense to leave it as 1 for the other Mixed Indices.
Just wondering if anyone else has tackled this for heftier mixed indices. In my particular use case it's largely being inflated due to wanting to leverage the full-text search over text documents that are themselves stored as vertex properties in my graph.
I think what I'd need to do would be:
1. Set the backing ES indices as read only (so make sure any JG uses at that time aren't going to try to mutate, maybe best to just turn JG off)
2. Invoke the ES split API, giving the post split indices a novel name
3. After the split completes, drop the original ES Index and assign the post split ES indices an alias of the original ES index's name. Probably would want JG definitely off for this.
4. Turn JG back on
Ideally this problem would be handled using ILM with an alias that JG expects and have policy in place that'd automatically roll the indices as they grow, but given only the latest index instance is allowed to receive writes though I think will cause problems if JG tries to mutate a document in a prior ES index since JG doesn't know anything about the index rolling so it won't be able to send the mutate command to particular instance through the Index API.
But it got me wondering if the name that JG presumably expects is an actual index is silently converted to an alias without its awareness would also cause issues?
Configuration wise it seems you're only able to convey to JanusGraph a singular primary shard & replica count when it creates an ES Index.
I'm projecting to eventually have a couple Mixed Indices exceed what will be reasonable for a single primary shard in the backing ES Index (given the rule of thumb of 10-50GB or 200M documents). So as a configuration default it makes sense to leave it as 1 for the other Mixed Indices.
Just wondering if anyone else has tackled this for heftier mixed indices. In my particular use case it's largely being inflated due to wanting to leverage the full-text search over text documents that are themselves stored as vertex properties in my graph.
I think what I'd need to do would be:
1. Set the backing ES indices as read only (so make sure any JG uses at that time aren't going to try to mutate, maybe best to just turn JG off)
2. Invoke the ES split API, giving the post split indices a novel name
3. After the split completes, drop the original ES Index and assign the post split ES indices an alias of the original ES index's name. Probably would want JG definitely off for this.
4. Turn JG back on
Ideally this problem would be handled using ILM with an alias that JG expects and have policy in place that'd automatically roll the indices as they grow, but given only the latest index instance is allowed to receive writes though I think will cause problems if JG tries to mutate a document in a prior ES index since JG doesn't know anything about the index rolling so it won't be able to send the mutate command to particular instance through the Index API.
But it got me wondering if the name that JG presumably expects is an actual index is silently converted to an alias without its awareness would also cause issues?
