Novu•3y ago

ReplyError: CROSSSLOT Keys in request don't hash to the same slot

Hi all,

We recently upgraded to the latest Novu release (we were stuck to the last release without workers so far) and we're noticing the following error in worker's logs which I can understand.

I don't know if that the reason but emails are being created using the API trigger endpoint, however we can't see anything in activity feed.

Any help would be appreciated

ReplyError: CROSSSLOT Keys in request don't hash to the same slot    at parseError (/usr/src/app/node_modules/.pnpm/redis-parser@3.0.0/node_modules/redis-parser/lib/parser.js:179:12)    at parseType (/usr/src/app/node_modules/.pnpm/redis-parser@3.0.0/node_modules/redis-parser/lib/parser.js:302:14) {  command: {    name: 'evalsha',    args: [      '3434c9fbf5ca138716481f924c25ce9705dc2a61',      '9',      'bull:metric:wait',      'bull:metric:active',      'bull:metric:priority',      'bull:metric:events',      'bull:metric:stalled',      'bull:metric:limiter',      'bull:metric:delayed',      'bull:metric:paused',      'bull:metric:meta',      'bull:metric:',      '1695805536211',      '',      <Buffer de 00 03 a5 74 6f 6b 65 6e d9 26 62 36 38 35 39 62 64 39 2d 33 63 38 63 2d 34 37 63 37 2d 61 31 62 38 2d 39 34 32 35 38 61 35 34 65 64 63 32 3a 30 ac ... 24 more bytes>    ]  }}

ReplyError: CROSSSLOT Keys in request don't hash to the same slot    at parseError (/usr/src/app/node_modules/.pnpm/redis-parser@3.0.0/node_modules/redis-parser/lib/parser.js:179:12)    at parseType (/usr/src/app/node_modules/.pnpm/redis-parser@3.0.0/node_modules/redis-parser/lib/parser.js:302:14) {  command: {    name: 'evalsha',    args: [      '3434c9fbf5ca138716481f924c25ce9705dc2a61',      '9',      'bull:metric:wait',      'bull:metric:active',      'bull:metric:priority',      'bull:metric:events',      'bull:metric:stalled',      'bull:metric:limiter',      'bull:metric:delayed',      'bull:metric:paused',      'bull:metric:meta',      'bull:metric:',      '1695805536211',      '',      <Buffer de 00 03 a5 74 6f 6b 65 6e d9 26 62 36 38 35 39 62 64 39 2d 33 63 38 63 2d 34 37 63 37 2d 61 31 62 38 2d 39 34 32 35 38 61 35 34 65 64 63 32 3a 30 ac ... 24 more bytes>    ]  }}

ReplyError: CROSSSLOT Keys in request don't hash to the same slot    at parseError (/usr/src/app/node_modules/.pnpm/redis-parser@3.0.0/node_modules/redis-parser/lib/parser.js:179:12)    at parseType (/usr/src/app/node_modules/.pnpm/redis-parser@3.0.0/node_modules/redis-parser/lib/parser.js:302:14) {  command: {    name: 'evalsha',    args: [      '3434c9fbf5ca138716481f924c25ce9705dc2a61',      '9',      'bull:metric:wait',      'bull:metric:active',      'bull:metric:priority',      'bull:metric:events',      'bull:metric:stalled',      'bull:metric:limiter',      'bull:metric:delayed',      'bull:metric:paused',      'bull:metric:meta',      'bull:metric:',      '1695805536211',      '',      <Buffer de 00 03 a5 74 6f 6b 65 6e d9 26 62 36 38 35 39 62 64 39 2d 33 63 38 63 2d 34 37 63 37 2d 61 31 62 38 2d 39 34 32 35 38 61 35 34 65 64 63 32 3a 30 ac ... 24 more bytes>    ]  }}

ReplyError: CROSSSLOT Keys in request don't hash to the same slot    at parseError (/usr/src/app/node_modules/.pnpm/redis-parser@3.0.0/node_modules/redis-parser/lib/parser.js:179:12)    at parseType (/usr/src/app/node_modules/.pnpm/redis-parser@3.0.0/node_modules/redis-parser/lib/parser.js:302:14) {  command: {    name: 'evalsha',    args: [      '3434c9fbf5ca138716481f924c25ce9705dc2a61',      '9',      'bull:metric:wait',      'bull:metric:active',      'bull:metric:priority',      'bull:metric:events',      'bull:metric:stalled',      'bull:metric:limiter',      'bull:metric:delayed',      'bull:metric:paused',      'bull:metric:meta',      'bull:metric:',      '1695805536211',      '',      <Buffer de 00 03 a5 74 6f 6b 65 6e d9 26 62 36 38 35 39 62 64 39 2d 33 63 38 63 2d 34 37 63 37 2d 61 31 62 38 2d 39 34 32 35 38 61 35 34 65 64 63 32 3a 30 ac ... 24 more bytes>    ]  }}

rolkool•9/27/23, 10:57 AM

Need to mention (in the same project with @serafeimo ) that we connect to a Redis Cluster in AWS though Elasticache Redis cluster

This is what we have in k8s:

            - name: REDIS_HOST
              value: >-
                my-develop-redis.ev1234
.clustercfg.euw2.cache.amazonaws.com
            - name: REDIS_PORT
              value: '6379'
            - name: REDIS_DB_INDEX
              value: '2'
            - name: REDIS_CACHE_SERVICE_HOST
              value: >-
                my-develop-redis.ev1234.clustercfg.euw2.cache.amazonaws.com
            - name: REDIS_CACHE_SERVICE_PORT
              value: '6379'
            - name: REDIS_CLUSTER_SERVICE_HOST
              value: >-
                my-develop-redis.ev1234.clustercfg.euw2.cache.amazonaws.com
            - name: REDIS_CLUSTER_SERVICE_PORTS
              value: '[6379]'
            - name: IS_IN_MEMORY_CLUSTER_MODE_ENABLED
              value: 'true'
            - name: IN_MEMORY_CLUSTER_MODE_ENABLED
              value: 'true'
            - name: REDIS_CLUSTER_DB_INDEX
              value: '2'

            - name: REDIS_HOST
              value: >-
                my-develop-redis.ev1234
.clustercfg.euw2.cache.amazonaws.com
            - name: REDIS_PORT
              value: '6379'
            - name: REDIS_DB_INDEX
              value: '2'
            - name: REDIS_CACHE_SERVICE_HOST
              value: >-
                my-develop-redis.ev1234.clustercfg.euw2.cache.amazonaws.com
            - name: REDIS_CACHE_SERVICE_PORT
              value: '6379'
            - name: REDIS_CLUSTER_SERVICE_HOST
              value: >-
                my-develop-redis.ev1234.clustercfg.euw2.cache.amazonaws.com
            - name: REDIS_CLUSTER_SERVICE_PORTS
              value: '[6379]'
            - name: IS_IN_MEMORY_CLUSTER_MODE_ENABLED
              value: 'true'
            - name: IN_MEMORY_CLUSTER_MODE_ENABLED
              value: 'true'
            - name: REDIS_CLUSTER_DB_INDEX
              value: '2'

Any idea how to avoid this error?

serafeimoOP•9/29/23, 9:05 AM

According to this page https://hackernoon.com/resolving-the-crossslot-keys-error-with-redis-cluster-mode-enabled

the problem could be solved by adding a prefix in the keys. Does the REDIS_CLUSTER_KEY_PREFIX property have to do with the above issue?

Resolving the CROSSSLOT Keys Error with Redis Cluster-Mode Enabled ...

Why "CROSSSLOT Keys in request don't hash to the same slot" error happens in a Redis Cluster, and how to solve it.

Pawan Jain•10/2/23, 12:47 PM

@rolkool @serafeimo
Are you running novu in k8s?

rolkool•10/4/23, 5:03 PM

Yes we do

serafeimoOP•10/6/23, 10:37 AM

From a quick investigation we had it seems that when Novu runs with Redis in cluster mode, it fails because it searches for the Redis host property (which is the single host). If we set this property in order to proceed, then it fails to send the keys due to the above reason (we suspect because it uses the single host with priority for some reason). I don't know if what I described makes sense but we haven't managed to find a solution apart from changing back to a single host.

Do you know if the cluster mode is tested and working?

serafeimoOP•10/23/23, 9:52 AM

Hi all! Any feedback on the above?

Sserafeimo From a quick investigation we had it seems that when Novu runs with Redis in clu...

--•10/23/23, 12:02 PM

Hello. Yes, the cluster mode is tested and working right now in our Cloud environments.
I am having a suspicion considered our code and our setup.
May I guess that you are using Elasticache for both the Redis Cache service and for the Redis Cluster service? Either 2 different instances or the same for both.
Can you confirm my guess?

--•10/23/23, 12:07 PM

Also can you confirm you are talking about v.0.19.0.

- name: REDIS_HOST value: >- my-develop-redis.ev1234 .clustercfg.euw2.cache.amazonaws.com - name: REDIS_PORT value: '6379' - name: REDIS_DB_INDEX value: '2' - name: REDIS_CACHE_SERVICE_HOST value: >- my-develop-redis.ev1234.clustercfg.euw2.cache.amazonaws.com - name: REDIS_CACHE_SERVICE_PORT value: '6379' - name: REDIS_CLUSTER_SERVICE_HOST value: >- my-develop-redis.ev1234.clustercfg.euw2.cache.amazonaws.com - name: REDIS_CLUSTER_SERVICE_PORTS value: '[6379]' - name: IS_IN_MEMORY_CLUSTER_MODE_ENABLED value: 'true' - name: IN_MEMORY_CLUSTER_MODE_ENABLED value: 'true' - name: REDIS_CLUSTER_DB_INDEX value: '2'

ReplyError: CROSSSLOT Keys in request don't hash to the same slot

Similar Threads

Similar Threads

Similar Threads