AWS RedisCluster not working / get master node failed about aredis HOT 8 CLOSED

I tried that out! It worked!

from aredis.

@kstrempel
Thanks.
It seems like that node_manager.initialize get no slots info when called, i will compare with redis-py-cluster code to find the real problem.

from aredis.

@kstrempel
Sorry to bring you inconvenience.
Since you don't refer the version of aredis, i suppose it to be the newest version(v1.1.1)
This problem seems to be caused by setting skip_full_coverage_check to True(it's surely correct considering the aws env)

I can't reproduce the problem in my dev env so it is hard for me to locate the bug accurately.
I can only see there may be something wrong when node manager of the connection pool is initialized. (connection pool needs to be initialized explicitly or initialized when calling execute_command implicitly, which is one difference between redis-py-cluster and aredis.) And then the slots dict of node manager is not initialized as expected.

If you don't mind, could you help me add log about the redis.connection_pool.initialized and redis.connection_pool.nodes.slots around your code please? Or could you run the script below in your env?

#!/usr/bin/python
# -*- coding: utf-8 -*-

import asyncio
import logging
from aredis import StrictRedisCluster


async def example():
    redis = StrictRedisCluster(startup_nodes=[{'host': '127.0.0.1', 'port': 7001}], decode_responses=False, skip_full_coverage_check=True)
    logging.info(redis.connection_pool.initialized)
    logging.info(redis.connection_pool.nodes.slots.keys())
    logging.info(await redis.get('a'))
    logging.info(redis.connection_pool.initialized)
    logging.info(redis.connection_pool.nodes.slots.keys())


if __name__ == '__main__':
    logging.basicConfig(level=logging.INFO)
    loop = asyncio.get_event_loop()
    loop.run_until_complete(example())

In a ideal empty cluster(for example, docker redis cluster in my dev env), output should like below

INFO:root:False
INFO:root:dict_keys([])
INFO:root:None
INFO:root:True
INFO:root:dict_keys([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, ..., 16383])

Looking forward for your response.

from aredis.

Hello,

sorry for the late anser. I run your small testscript against the AWS redis cluster.

This is the result

INFO:root:False
INFO:root:dict_keys([])
Traceback (most recent call last):
  File "testredis.py", line 21, in <module>
    loop.run_until_complete(example())
  File "/usr/local/lib/python3.6/asyncio/base_events.py", line 466, in run_until_complete
    return future.result()
  File "testredis.py", line 13, in example
    logging.info(await redis.get('a'))
  File "/usr/local/lib/python3.6/site-packages/aredis/commands/strings.py", line 147, in get
    return await self.execute_command('GET', name)
  File "/usr/local/lib/python3.6/site-packages/aredis/utils.py", line 167, in inner
    return await func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/aredis/client.py", line 405, in execute_command
    node = self.connection_pool.get_node_by_slot(slot)
  File "/usr/local/lib/python3.6/site-packages/aredis/pool.py", line 480, in get_node_by_slot
    return self.get_master_node_by_slot(slot)
  File "/usr/local/lib/python3.6/site-packages/aredis/pool.py", line 475, in get_master_node_by_slot
    return self.nodes.slots[slot][0]
KeyError: 15495

I had to change the connection part a bit.

async def example():
    redis = StrictRedisCluster(startup_nodes=[{'host': 'redis', 'port': 6379}], decode_responses=False, skip_full_coverage_check=True)
    logging.info(redis.connection_pool.initialized)
    logging.info(redis.connection_pool.nodes.slots.keys())
    logging.info(await redis.get('a'))
    logging.info(redis.connection_pool.initialized)
    logging.info(redis.connection_pool.nodes.slots.keys())

A AWS you have no 7000, 7001 port for the cluster.

from aredis.

@kstrempel
Thanks for your feedback~
Please try code below to see if this problem is caused by not calling connection_pool.initialize or slots not obtained in nodemanager.initialize

#!/usr/bin/python
# -*- coding: utf-8 -*-

import asyncio
import logging
from aredis import StrictRedisCluster


async def example():
    redis = StrictRedisCluster(startup_nodes=[{'host': '127.0.0.1', 'port': 7001}], decode_responses=False, skip_full_coverage_check=True)
    logging.info(redis.connection_pool.initialized)
    logging.info(redis.connection_pool.nodes.slots.keys())
    await redis.connection_pool.initialize()
    logging.info(redis.connection_pool.initialized)
    logging.info(redis.connection_pool.nodes.slots[15495])
    logging.info(await redis.get('a'))


if __name__ == '__main__':
    logging.basicConfig(level=logging.INFO)
    loop = asyncio.get_event_loop()
    loop.run_until_complete(example())

from aredis.

Here it is.
I had to change the connection part again to 'port': 6397.

INFO:root:False
INFO:root:dict_keys([])
INFO:root:True
Traceback (most recent call last):
  File "test.py", line 22, in <module>
    loop.run_until_complete(example())
  File "/usr/local/lib/python3.6/asyncio/base_events.py", line 466, in run_until_complete
    return future.result()
  File "test.py", line 15, in example
    logging.info(redis.connection_pool.nodes.slots[15495])
KeyError: 15495

BTW I forgot to mention that our current solution is running with the redis-py-cluster==1.3.4 package, with the exact same connection configuration. Perhaps that helps to find the problem.

from aredis.

@kstrempel
Could you help to try the code optimized? I compared this part of code with redis-py-cluster, most logic of them is the same but the process of single node situation.
But i am not sure if it is possible that you have only one node in your aws redis cluster.
you can install the newest version by:
pip install git+https://github.com/NoneGG/aredis
If that still not work, could you show me your cluster slot info please?

from aredis.

@vineetgoel
Thanks a lot sincerely, that really helps!
@kstrempel
If you don't mind, this issue will be closed, thank you again for you bug report~
If there are still problems, please reopen the issue.

from aredis.