MergeV "get or create" performance asymmetry
So I'm working on adding the mergeV step among others to the Rust gremlin driver. As part of that I took a pause and did a performance comparison to the "traditional" way of doing it.
So the Rust driver is submitting bytecode that's effectively doing:
"Traditional/Reference":
^ But given a batch of 10k vertices to write it'd do this for a chunk of 10 vertices in a single mutation traversal, but doing 10 connections in parallel to split up the batch until it finished getting all 10k written. It's well known that very long traversals don't perform well and my own trials found that doing this at > 50 vertices in a single traversal would cause timeouts for my use case, so I've been generally doing 10 and calling it good.
But this puts a ceiling on the amount of work a single network call can make (10 vertices worth) so hence why I started trying out
And then the "mergeV()" way:
I would run the mergeV call with chunks of 200 vertices in each call.
So the Rust driver is submitting bytecode that's effectively doing:
"Traditional/Reference":
^ But given a batch of 10k vertices to write it'd do this for a chunk of 10 vertices in a single mutation traversal, but doing 10 connections in parallel to split up the batch until it finished getting all 10k written. It's well known that very long traversals don't perform well and my own trials found that doing this at > 50 vertices in a single traversal would cause timeouts for my use case, so I've been generally doing 10 and calling it good.
But this puts a ceiling on the amount of work a single network call can make (10 vertices worth) so hence why I started trying out
mergeV()And then the "mergeV()" way:
I would run the mergeV call with chunks of 200 vertices in each call.
