JanusGraph fails on initial startup (cql-es)
Hi all,
I have a problem that's been popping up once in a while regarding only the initial startup of Cassandra/ES/JanusGraph.
Sometimes it fails with "table edgestore doesn't exist", or "table janusgraph_ids doesn't exist". However, if I restart the JG pod, it completes. To me, it sounds like some sort of race condition.
There's a thread for JG 0.6.0 with the same error:
https://lists.lfaidata.foundation/g/janusgraph-users/topic/unconfigured_table_exceptions/85741983?p=,,,20,0,0,0::recentpostdate/sticky,,,20,2,0,85741983,previd=1634849714017760765,nextid=1631631322121685927&previd=1634849714017760765&nextid=1631631322121685927
I'm just surprised it still exists? Is there a good fix (I don't think the thread above gives a consistent solution)?
For reference, I'm using:
janusgraph/janusgraph:1.0.0-20230805-060925.b584085
cassandra:4.0.10
docker.elastic.co/elasticsearch/elasticsearch:7.17.10
To make matters worse, this issue only seem to happen in low-resource hardware. I cannot reproduce locally on my M1 machine, but it happens consistently in a github actions-workflow that spins up a docker-compose with ES, Cassandra and JG.
I can share if interested.
3 Replies
Have you tried increasing advanced.metadata.schema.debouncer.window as described in that thread?
This is related to how CQL driver works for schema updates. You either make schema creation process faster by setting that config to 0 (i.e. default in JanusGraph), or you increase that setting which will make schema creation longer but will give other nodes enough time to fetch the new schema.
No honestly I hadn't. I struggled a bit with figuring out the formatting and which property to set, however it seems that adding it as:
As an environment config for JG resolves the issue
I'll try it out a couple of times over the next 24h 🙂
FWIW, It's always been a problem with Amazon keyspace, where table creation always takes a long time (around 10 seconds). The mitigation is simply to retry until all tables have been successfully built.