flyerhzm / redis-sentinel Goto Github PK

another redis automatic master/slave failover solution for ruby by using built-in redis sentinel (deprecated)

License: MIT License

Ruby 100.00%

redis-sentinel's Introduction

Redis::Sentinel (Deprecated)

The redis gem supports sentinel from version 3.2, redis-sentinel is not necessary if you are using Redis 2.8.x or later.

Another redis automatic master/slave failover solution for ruby by using built-in redis sentinel.

It subscribes message with channel "+switch-master", when message received, it will disconnect current connection and connect to new master server.

Installation

Add this line to your application's Gemfile:

gem 'redis-sentinel'

If you are using redis-server less than 2.6.10, please use redis-sentinel 1.3.0

gem 'redis-sentinel', '~> 1.3.0'

And then execute:

$ bundle

Or install it yourself as:

$ gem install redis-sentinel

Usage

Specify the sentinel servers and master name

Redis.new(master_name: "master1", sentinels: [{host: "localhost", port: 26379}, {host: "localhost", port: 26380}])

Sentinels can also be specified using a URI. This URI syntax is required when using Rails.config.cache_store:

config.cache_store = :redis_store, { master_name: "master1",
                                     sentinels: ['sentinel://localhost:26379', 'sentinel://localhost:26380'] }

After doing the above, you might still see #<Redis client v3.1.0 for redis://localhost:6379/0>. This is fine because redis-sentinel will only try to connect when it is actually required.

However, if none of the sentinel servers can be reached, a Redis::CannotConnectError will be thrown.

There are two additional options:

:failover_reconnect_timeout (seconds) will block for that long when redis is unreachable to give failover enough time to take place. Does not wait if not given, or time given is 0.
:failover_reconnect_wait (seconds) how long to sleep after each failed reconnect during a failover event. Defaults to 0.1s.

Slaves clients

If you need it, you can get an array of Redis clients, each pointing to one of the slaves:

client = Redis.new(master_name: "master1", sentinels: [{host: "localhost", port: 26379}, {host: "localhost", port: 26380}])
client.slaves
# => [#<Redis client v3.0.7 for redis://127.0.0.1:6380/0>, #<Redis client v3.0.7 for redis://127.0.0.1:6381/0>]

You can also get an array of all the clients (master + slaves):

client = Redis.new(master_name: "master1", sentinels: [{host: "localhost", port: 26379}, {host: "localhost", port: 26380}])
client.all_clients
# => [#<Redis client v3.0.7 for redis://127.0.0.1:6379/0>, #<Redis client v3.0.7 for redis://127.0.0.1:6380/0>, #<Redis client v3.0.7 for redis://127.0.0.1:6381/0>]

Example

Start redis master server, listen on port 16379

$ redis-server example/redis-master.conf

Start redis slave server, listen on port 16380

$ redis-server example/redis-slave.conf

Start 2 sentinel servers

$ redis-server example/redis-sentinel1.conf --sentinel
$ redis-server example/redis-sentinel2.conf --sentinel
$ redis-server example/redis-sentinel3.conf --sentinel

Run example/test.rb, which will query value of key "foo" every second.

$ bundle exec ruby example/test.rb

You will see output "bar" every second. Let's try the failover process.

Stop redis master server.
You will see error message output.
Redis sentinel promote redis slave server to master. During this time you will see errors instead of "bar" while the failover is happening.
Then you will see correct "bar" output every second again.

Example of Failover Timeout

Run the same example code above but run:

$ bundle exec ruby example/test_wait_for_failover.rb

You will see the stream of "bar" will stop while failover is taking place and will resume once it has completed, provided that failover takes less than 30 seconds.

Authors and Contributors

https://github.com/flyerhzm/redis-sentinel/graphs/contributors

Please fork and contribute, any help in making this project better is appreciated!

This project is a member of the OSS Manifesto.

Copyright

redis-sentinel's People

Contributors

Stargazers

Watchers

Forkers

dplummer royaltm zanker colinrymer magec patricktulskie ltvco nick-desteffen covermymeds yankov remy1991 tmeinlschmidt aaronrustad zendesk dim juneng603 excds eccyan sgonyea aniketschneider n1te1337 imclab rafaelgaspar vkhater gfriebe sipims krasnoukhov rlhh reedf taskrabbit liuhui998 zmack hujiko primary0 catawiki halfblood369 daniel-g 5470x3 rockabe phuongnd08 yhtsnda dejavu1219 mrkeng tvon delkyd hunterchen junli0411 chrisguorui zhoujia916 rahmatov wal-noc crowd9 yangxiyucs wu-wuxl sidd081-zz skymysky nynucyw quyetmv liv3010m ashokaupadhya imxty julianrazif jetli123

redis-sentinel's Issues

Bubble up error so that I can kill my server if there is no redis sentinel connection

If I have these two lines of code:

var Sentinel = sentinel.Sentinel(endpoints);
var client = Sentinel.createClient(REDIS_SENTINEL_MASTER_NAME, {role: 'master'});

When Redis is not running, or it is misconfigured, your library will throw an error - Error: Failed to find a sentinel from the endpoints

However, the error is not fatal and my server keeps running.

(1) I want to trap that error and decide if I should kill my server.
(2) I want to find out where in my program the error is occurring; as of now, it's hard to track down.

In most sophisticated languages, errors will "bubble up" if you enclose them with a try/catch block:

try{
    var Sentinel = sentinel.Sentinel(endpoints);
    var client = Sentinel.createClient(REDIS_SENTINEL_MASTER_NAME, {role: 'master'});
}
catch(e){
    throw e;
}

but I am unable to trap the errors. Your library throws the error internally, but doesn't give me enough information to trap the error.

Am I using the wrong approach?

Version 1.3

Would it be unreasonable to request a new gem version be released with the few changes since April?

Hi! There looks to be a bug with the underlying connections, in that if one of the sentinel hosts is unreachable (IE, the server is off), we still throw a No route to host - connect(2) error, rather than moving on and trying another sentinel host.

Round robin to redis slaves

We're using the redis sentinel to connect to our Redis cluster. We want to use node sentinel to connect to slave machines as:

           options =
                    no_ready_check: true
                    socket_keepalive: true
                    retry_max_delay: 3000
                    connect_timeout: 1000
                    max_attempts: 5
                    role: 'slave'
            redisClient = sentinel.createClient(sentinelConfigs, masterName, options)

Would that result in choosing a redis slave based on round robin fashion every time we call createClient(), or will always pick up the same redis slave in the cluster?

Add information that redis-sentinel should not be used with redis >= 3.2

We've had some problems after upgrading the redis gem while still using the redis-sentinel. It broke the newly added sentinel support on redis-rb 3.2 and didn't raise any errors. We only found out when deploying to production and the redis client was incorrectly attempting to connect to the localhost.

I'd love to see a warning on the readme or something, so this can be avoided by others in the future :)

Reconnects broken since 1.4

In our staging setup, we have a single sentinel running. On the first redis timeout/disconnect, we start to receive continuous Redis::CannotConnectError errors. The client doesn't seem to reconnect an the the persist on every redis call until we restart the app.

I think there is a problem with try_next_sentinel, as it shifts from the list of @sentinel_options, never returning them back to the array. I didn't dive into the code, but shouldn't the method look something like this:

def try_next_sentinel
  sentinel_options = @sentinels_options.shift
  if sentinel_options
    @logger.debug "Trying next sentinel: #{sentinel_options[:host]}:#{sentinel_options[:port]}" if @logger && @logger.debug?
    @current_sentinel_options = sentinel_options
    @current_sentinel = Redis.new sentinel_options
  else
    raise Redis::CannotConnectError
  end
ensure
   @sentinels_options.push(sentinel_options) if sentinel_options
end

Full backtrace:

redis-sentinel-1.4.0/lib/redis-sentinel/client.rb:61→ try_next_sentinel
redis-sentinel-1.4.0/lib/redis-sentinel/client.rb:74→ discover_master
redis-sentinel-1.4.0/lib/redis-sentinel/client.rb:28→ block in connect_with_sentinel
redis-sentinel-1.4.0/lib/redis-sentinel/client.rb:46→ call
redis-sentinel-1.4.0/lib/redis-sentinel/client.rb:46→ auto_retry_with_timeout
redis-sentinel-1.4.0/lib/redis-sentinel/client.rb:27→ connect_with_sentinel
redis-3.0.6/lib/redis/client.rb:292→ ensure_connected

Failure to define :sentinels option results in error when calling 'slaves'

I realize this is an edge case, but in development I have no sentinels so left off the :sentinels option entirely. For kicks I then called slaves on my redis object and got this error:

NoMethodError: undefined method `shift' for nil:NilClass
from .../gems/redis-sentinel-1.5.0/lib/redis-sentinel/client.rb:56:in `try_next_sentinel'

From reading the source I think tweaking parse_sentinel_options to return an empty array if nothing is provided would fix it.

See: https://github.com/flyerhzm/redis-sentinel/blob/master/lib/redis-sentinel/client.rb#L201

I'm happy to make the change and add specs if you agree this is the right behavior.

sentinel failover issue

While redis sentinel is performing failover procedure it returns newly promoted master address (as a response from get-master-addr-by-name) BEFORE actually the promoted slave BECOMES a MASTER.

When you've set slave-serve-stale-data to 'no' on your redis-servers, then just before the failover procedure ends the following error occurs in redis client (with redis-sentinel patch):

Redis::CommandError MASTERDOWN Link with MASTER is down and slave-serve-stale-data is set to 'no'.

instead of Redis::CannotConnectError which is being rescued in #auto_retry_with_timeout().

To recreate the error:

Setup your redis servers (2.6.9):

redis1.conf:
daemonize yes
pidfile /var/run/redis/redis1.pid
port 6380
bind 127.0.0.1
slave-serve-stale-data no
# everything else by default

redis2.conf:
daemonize yes
pidfile /var/run/redis/redis2.pid
port 6381
bind 127.0.0.1
slave-serve-stale-data no
# everything else by default

sentinel.conf:
sentinel monitor mymaster 127.0.0.1 6380 1
sentinel down-after-milliseconds mymaster 2000
sentinel can-failover mymaster yes
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 900000

Now run:

redis-server redis1.conf
redis-server redis2.conf --slaveof 127.0.0.1 6380
redis-server sentinel.conf --sentinel

The case:

require 'redis'
require 'redis-sentinel'

r = Redis.new(master_name: 'mymaster',
    sentinels: [{host:'127.0.0.1',port:26379}],
    failover_reconnect_timeout: 50,
    failover_reconnect_wait: 0.01)

puts 'set foo'
puts r.set :foo, 1
sleep 1
puts 'shutdown'
r.shutdown
sleep 1
puts 'get foo'
puts r.get :foo

The result:

set foo
OK
shutdown
get foo
/usr/local/lib/ruby/gems/1.9.1/gems/redis-3.0.2/lib/redis/client.rb:85:in `call': MASTERDOWN Link with MASTER is down and slave-serve-stale-data is set to 'no'. (Redis::CommandError)
        from /usr/local/lib/ruby/gems/1.9.1/gems/redis-3.0.2/lib/redis.rb:715:in `block in get'
        from /usr/local/lib/ruby/gems/1.9.1/gems/redis-3.0.2/lib/redis.rb:36:in `block in synchronize'
        from /usr/local/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
        from /usr/local/lib/ruby/gems/1.9.1/gems/redis-3.0.2/lib/redis.rb:36:in `synchronize'
        from /usr/local/lib/ruby/gems/1.9.1/gems/redis-3.0.2/lib/redis.rb:714:in `get'
        from -:16:in `<main>'

When you set :failover_reconnect_wait to let's say 0.5 you might actually miss the error, but it WILL hit you on occasion, just not every time like in the example above.

The quick fix is to patch #auto_retry_with_timeout:

    def auto_retry_with_timeout(&block)
      deadline = @failover_reconnect_timeout.to_i + Time.now.to_f
      begin
        block.call
      rescue Redis::CannotConnectError, Redis::CommandError => e
        raise unless Redis::CannotConnectError === e || \
          /MASTERDOWN Link with MASTER is down/ === e.message
        raise if Time.now.to_f > deadline
        sleep @failover_reconnect_wait
        retry
      end
    end

Another workaround is to set slave-serve-stale-data to yes, but the problem will probably arise when you try to issue a writing command to slave with slave-read-only set to yes.

I think the real issue is with the senitnel redis-server. It just shouldn't inform about new master before ensuring its reconfiguration. However this is the reality and since the redis-sentinel project is somewhat abandoned in favor of redis-cluster we have to deal with it here.

Subscribe to switch master message is necessary

If there is a false alarm, and the sentinels promote a slave to master even the master is still alive, the client will keep connected to the old master, which becomes slave as a result of the switch.

@flyerhzm Why was the code that subscribe to the +switch-master event removed and can we bring it back?

Inconsistent failover when there is no failover timeout set

This bug occurs if there is no :failover_reconnect_timeout configured and a failover occurs, but the client is still able to connect to the old master node (which is now a slave). The client will stay connected to the old master node and writes will fail with READONLY errors.

This situation can be easily replicated by setting up sentinels as in the example, but changing the example script to write instead of just reading. Run the example script, then issue sentinel failover example-test to one of the sentinel nodes.

My take on the bug is as follows:

when the client can't connect at all to the server, a ConnectionError is raised and a reconnect is initiated (via code from the redis-rb gem).
when the client attempts a write and can connect to the server, but the server is a READONLY slave, a CommandError is raised instead.
if there is no reconnect timeout configured, or the reconnect timeout is 0, this check raises the error back to the base redis client without attempting to reconnect
because it is a CommandError, the code in the redis-rb gem does not try to reconnect, so the client remains connected to the slave.

Basically if you don't have a failover_reconnect_timeout configured, or if you have it configured to 0, the client gives up attempting to reconnect without trying to reconnect even once.

Stacktrace due to offline sentinel during reconnect

Hi,

Assuming a Redis Master -> Slave setup with two sentinels; if the sentinel that the redis-sentinel gem is currently connected to goes offline, the next call to client.reconnect will cause a stacktrace like the following:

[+1.00096] COUNTER:    61
[+1.00137] COUNTER:    62
[+1.00129] COUNTER:    63
[+1.00143] COUNTER:    64
[+1.00147] COUNTER:    65
RECONNECT
[+1.00171] COUNTER:    66
[+1.00168] COUNTER:    67
[+1.00137] COUNTER:    68
[+1.00144] COUNTER:    69
[+1.00118] COUNTER:    70
RECONNECT
/usr/local/stow/ruby-1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:276:in `rescue in establish_connection': Error connecting to     Redis on 192.168.201.47:26379 (ECONNREFUSED) (Redis::CannotConnectError)
    from /usr/local/stow/ruby-1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:271:in `establish_connection'
    from /usr/local/stow/ruby-1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:69:in `connect'
    from /usr/local/stow/ruby-1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-sentinel-1.3.0/lib/redis-sentinel/client.rb:29:in `connect_with_sentinel'
from /usr/local/stow/ruby-1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:202:in `reconnect'
from /usr/local/stow/ruby-1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-sentinel-1.3.0/lib/redis-sentinel/client.rb:96:in `reconnect_with_sentinels'
from /usr/local/stow/ruby-1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-sentinel-1.3.0/lib/redis-sentinel/client.rb:94:in `block in reconnect_with_sentinels'
from /usr/local/stow/ruby-1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-sentinel-1.3.0/lib/redis-sentinel/client.rb:93:in `each'
from /usr/local/stow/ruby-1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-sentinel-1.3.0/lib/redis-sentinel/client.rb:93:in `reconnect_with_sentinels'
from ./redis-ha-test.rb:25:in `<main>'

License missing from gemspec

RubyGems.org doesn't report a license for your gem. This is because it is not specified in the gemspec of your last release.

via e.g.

spec.license = 'MIT'
# or
spec.licenses = ['MIT', 'GPL-2']

Including a license in your gemspec is an easy way for rubygems.org and other tools to check how your gem is licensed. As you can imagine, scanning your repository for a LICENSE file or parsing the README, and then attempting to identify the license or licenses is much more difficult and more error prone. So, even for projects that already specify a license, including a license in your gemspec is a good practice. See, for example, how rubygems.org uses the gemspec to display the rails gem license.

There is even a License Finder gem to help companies/individuals ensure all gems they use meet their licensing needs. This tool depends on license information being available in the gemspec. This is an important enough issue that even Bundler now generates gems with a default 'MIT' license.

I hope you'll consider specifying a license in your gemspec. If not, please just close the issue with a nice message. In either case, I'll follow up. Thanks for your time!

Appendix:

If you need help choosing a license (sorry, I haven't checked your readme or looked for a license file), GitHub has created a license picker tool. Code without a license specified defaults to 'All rights reserved'-- denying others all rights to use of the code.
Here's a list of the license names I've found and their frequencies

p.s. In case you're wondering how I found you and why I made this issue, it's because I'm collecting stats on gems (I was originally looking for download data) and decided to collect license metadata,too, and make issues for gemspecs not specifying a license as a public service :). See the previous link or my blog post about this project for more information.

undefined method `values_at' for nil:NilClass

I keep getting this error no matter what the config here is the stacktrace:
Stacktrace:

redis-sentinel/client.rb:65:in `refresh_sentinels_list'
    @sentinels_options.uniq! {|h| h.values_at(:host, :port) }
  org/jruby/RubyArray.java:3209:in `uniq!'
  redis-sentinel/client.rb:65:in `refresh_sentinels_list'
    @sentinels_options.uniq! {|h| h.values_at(:host, :port) }
  redis-sentinel/client.rb:77:in `discover_master'
    refresh_sentinels_list
  redis-sentinel/client.rb:27:in `connect_with_sentinel'
    discover_master
  org/jruby/RubyProc.java:271:in `call'
  redis-sentinel/client.rb:45:in `auto_retry_with_timeout'
    block.call
  redis-sentinel/client.rb:26:in `connect_with_sentinel'
    auto_retry_with_timeout do
  redis/client.rb:304:in `ensure_connected'
    connect
  redis/client.rb:191:in `process'
    ensure_connected do
  redis/client.rb:270:in `logging'
    return yield unless @logger && @logger.debug?
  redis/client.rb:190:in `process'
    logging(commands) do
  redis/client.rb:161:in `call_pipelined'
    process(commands) do
  redis/client.rb:133:in `call_pipeline'
    pipeline.finish(call_pipelined(pipeline.commands)).tap do
  redis/client.rb:257:in `with_reconnect'
    yield
  redis/client.rb:131:in `call_pipeline'
    with_reconnect pipeline.with_reconnect? do
  redis-sentinel/client.rb:145:in `readonly_protection_with_timeout'
    send(method, *args, &block)
  redis-sentinel/client.rb:136:in `call_pipeline_with_readonly_protection'
    readonly_protection_with_timeout(:call_pipeline_without_readonly_protection, *args, &block)
  redis.rb:2094:in `multi'
    original.call_pipeline(pipeline)
  redis.rb:37:in `synchronize'
    mon_synchronize { yield(@client) }
  monitor.rb:211:in `mon_synchronize'
    yield
  monitor.rb:210:in `mon_synchronize'
    begin
  redis.rb:37:in `synchronize'
    mon_synchronize { yield(@client) }
  redis.rb:2086:in `multi'
    synchronize do |client|
  redis/namespace.rb:413:in `namespaced_block'
    redis.send(command) do |r|
  redis/namespace.rb:258:in `multi'
    namespaced_block(:multi, &block)
  sidekiq/client.rb:180:in `raw_push'
    conn.multi do
  connection_pool.rb:58:in `with'
    yield conn
  sidekiq/client.rb:179:in `raw_push'
    @redis_pool.with do |conn|
  sidekiq/client.rb:68:in `push'
    pushed = raw_push([payload]) if payload
  sidekiq/worker.rb:85:in `client_push'
    Sidekiq::Client.new(pool).push(item.stringify_keys)
  sidekiq/worker.rb:40:in `perform_async'
    client_push('class' => self, 'args' => args)
  devise/async/backend/sidekiq.rb:10:in `enqueue'
    perform_async(*args)
  devise/async/worker.rb:11:in `enqueue'
    backend_class.enqueue(method, resource_class, resource_id, *args)
  devise/async/model.rb:37:in `send_devise_notification'
    Devise::Async::Worker.enqueue(notification, self.class.name, self.id.to_s, *args)
  devise_invitable/model.rb:168:in `deliver_invitation'
    send_devise_notification(:invitation_instructions)
  devise_invitable/model.rb:132:in `invite!'
    deliver_invitation unless @skip_invitation
  devise_invitable/model.rb:240:in `_invite'
    mail = invitable.invite!
  devise_invitable/model.rb:252:in `invite!'
    invitable, mail = _invite(attributes, invited_by, &block)
  devise/invitations_controller.rb:16:in `create'
    self.resource = resource_class.invite!(resource_params, current_inviter)
  devise/invitations_controller.rb:32:in `create'
    super
  action_controller/metal/implicit_render.rb:4:in `send_action'
    ret = super
  abstract_controller/base.rb:167:in `process_action'
    send_action(method_name, *args)
  action_controller/metal/rendering.rb:10:in `process_action'
    super
  abstract_controller/callbacks.rb:18:in `process_action'
    super
  active_support/callbacks.rb:472:in `_run__1802481893__process_action__984953711__callbacks'
    # When creating or skipping callbacks, you can specify conditions that
  active_support/callbacks.rb:405:in `__run_callback'
    object.send(name, &blk)
  active_support/callbacks.rb:390:in `_run_process_action_callbacks'
    end
  active_support/callbacks.rb:81:in `run_callbacks'
    send("_run_#{kind}_callbacks", *args, &block)
  abstract_controller/callbacks.rb:17:in `process_action'
    run_callbacks(:process_action, action_name) do
  mongoid/unit_of_work.rb:39:in `unit_of_work'
    yield if block_given?
  mongoid/unit_of_work.rb:37:in `unit_of_work'
    begin
  rack/mongoid/middleware/identity_map.rb:34:in `call'
    ::Mongoid.unit_of_work { @app.call(env) }
  warden/manager.rb:35:in `call'
    @app.call(env)
  org/jruby/RubyKernel.java:1282:in `catch'
  warden/manager.rb:34:in `call'
    result = catch(:warden) do
  action_dispatch/middleware/best_standards_support.rb:17:in `call'
    status, headers, body = @app.call(env)
  rack/etag.rb:23:in `call'
    status, headers, body = @app.call(env)
  rack/conditionalget.rb:35:in `call'
    @app.call(env)
  action_dispatch/middleware/head.rb:14:in `call'
    @app.call(env)
  action_dispatch/middleware/params_parser.rb:21:in `call'
    @app.call(env)
  action_dispatch/middleware/flash.rb:242:in `call'
    @app.call(env)
  rack/session/abstract/id.rb:210:in `context'
    status, headers, body = app.call(env)
  rack/session/abstract/id.rb:205:in `call'
    context(env)
  action_dispatch/middleware/cookies.rb:341:in `call'
    status, headers, body = @app.call(env)
  action_dispatch/middleware/callbacks.rb:28:in `call'
    @app.call(env)
  active_support/callbacks.rb:408:in `_run__1990414355__call__699693046__callbacks'
    def __reset_runner(symbol)
  active_support/callbacks.rb:405:in `__run_callback'
    object.send(name, &blk)
  active_support/callbacks.rb:390:in `_run_call_callbacks'
    end
  active_support/callbacks.rb:81:in `run_callbacks'
    send("_run_#{kind}_callbacks", *args, &block)
  action_dispatch/middleware/callbacks.rb:27:in `call'
    run_callbacks :call do
  action_dispatch/middleware/remote_ip.rb:31:in `call'
    @app.call(env)
  action_dispatch/middleware/debug_exceptions.rb:16:in `call'
    response = @app.call(env)
  action_dispatch/middleware/show_exceptions.rb:56:in `call'
    response  = @app.call(env)
  rails/rack/logger.rb:32:in `call_app'
    @app.call(env)
  rails/rack/logger.rb:16:in `call'
    Rails.logger.tagged(compute_tags(request)) { call_app(request, env) }
  active_support/tagged_logging.rb:22:in `tagged'
    yield self
  rails/rack/logger.rb:16:in `call'
    Rails.logger.tagged(compute_tags(request)) { call_app(request, env) }
  action_dispatch/middleware/request_id.rb:22:in `call'
    status, headers, body = @app.call(env)
  rack/methodoverride.rb:21:in `call'
    @app.call(env)
  rack/runtime.rb:17:in `call'
    status, headers, body = @app.call(env)
  active_support/cache/strategy/local_cache.rb:72:in `call'
    @app.call(env)
  action_dispatch/middleware/static.rb:63:in `call'
    @app.call(env)
  rack/cache/context.rb:136:in `forward'
    Response.new(*backend.call(@env))
  rack/cache/context.rb:143:in `pass'
    forward
  rack/cache/context.rb:155:in `invalidate'
    pass
  rack/cache/context.rb:71:in `call!'
    invalidate
  rack/cache/context.rb:51:in `call'
    clone.call! env
  raven/rack.rb:59:in `call'
    response = @app.call(env)
  rails/engine.rb:484:in `call'
    app.call(env.merge!(env_config))
  rails/application.rb:231:in `call'
    super(env)
  rails/railtie/configurable.rb:30:in `method_missing'
    instance.send(*args, &block)
  puma/configuration.rb:68:in `call'
    @app.call(env)
  puma/server.rb:486:in `handle_request'
    status, headers, res_body = @app.call(env)
  puma/server.rb:484:in `handle_request'
    begin
  puma/server.rb:357:in `process_client'
    case handle_request(client, buffer)
  puma/server.rb:353:in `process_client'
    begin
  puma/server.rb:250:in `run'
    process_client client, buffer
  org/jruby/RubyProc.java:271:in `call'
  puma/thread_pool.rb:92:in `spawn_thread'
    block.call(work, *extra)

Here is what my config looks like:

require 'redis/namespace'

if Rails.env.production?
  # Using redis-sentinel gem to control sentinel layer
  # https://github.com/mperham/sidekiq/wiki/Advanced-Options
  options = {
    master_name: "master name",
    sentinels: [
      {host: "ip", port: "26379"}, 
      {host: "ip", port: "26379"}, 
      {host: "ip", port: "26379"} #  - arbiter
    ],
    failover_reconnect_timeout: 15, # give 15s for failover to take place
    failover_reconnect_wait: 10 # wait 10s after each failed reconnect
  }

  redis_conn = proc {
    # The following lines use a combination of redis-sentinel and
    # the redis namespace gems to give the connection pool the redis client object
    client = Redis.new(options)
    Redis::Namespace.new("bg_jobs", redis: client)
  }
  Sidekiq.configure_client do |config|
    config.redis = ConnectionPool.new(timeout: 10, size: 5, &redis_conn)
  end
  Sidekiq.configure_server do |config|
    config.redis = ConnectionPool.new(timeout: 10, size: 25, &redis_conn)
  end

else
  port = Rails.env.test? ? "9736" : "6379"
  options = {url: "redis://127.0.0.1:#{port}/12", namespace: "bg_jobs"}
  Sidekiq.configure_server do |config|
    config.redis = options
  end

  Sidekiq.configure_client do |config|
    config.redis = options
  end
end

It of course only happens in production, and I tried doing it all in the console and I get the same errors, however I can get all_clients and for some odd reason it works if I Redis.new().client.current_sentinel.

Any help explaining this would be really appreciated

Handle +sentinel messages

In an Core OS cluster, sentinels can be dynamically relocated to different machines during run time. Handle +sentinel (and also -sentinel) is important.

Problem with redis-sentinel + resque

Hi.
I am trying to start resque with redis-sentinel.
My config:
Resque.redis = Redis.new(master_name: "localhost", sentinels: [{host: "localhost", port: 26379}, {host: "localhost", port: 26380}])

When I start resque then I get error:

too large fdsets
/opt/web-collaboration/shared/bundle/ruby/1.9.1/gems/redis-3.0.2/lib/redis/connection/hiredis.rb:16:in connect' /opt/web-collaboration/shared/bundle/ruby/1.9.1/gems/redis-3.0.2/lib/redis/connection/hiredis.rb:16:inconnect'
/opt/web-collaboration/shared/bundle/ruby/1.9.1/gems/redis-3.0.2/lib/redis/client.rb:263:in establish_connection' /opt/web-collaboration/shared/bundle/ruby/1.9.1/gems/redis-3.0.2/lib/redis/client.rb:69:inconnect'
/opt/web-collaboration/shared/bundle/ruby/1.9.1/gems/redis-sentinel-1.1.2/lib/redis-sentinel/client.rb:16:in connect_with_sentinel' /opt/web-collaboration/shared/bundle/ruby/1.9.1/gems/redis-3.0.2/lib/redis/client.rb:282:inensure_connected'
/opt/web-collaboration/shared/bundle/ruby/1.9.1/gems/redis-3.0.2/lib/redis/client.rb:173:in block in process' /opt/web-collaboration/shared/bundle/ruby/1.9.1/gems/redis-3.0.2/lib/redis/client.rb:248:inlogging'
/opt/web-collaboration/shared/bundle/ruby/1.9.1/gems/redis-3.0.2/lib/redis/client.rb:172:in process' /opt/web-collaboration/shared/bundle/ruby/1.9.1/gems/redis-3.0.2/lib/redis/client.rb:84:incall'
/opt/web-collaboration/shared/bundle/ruby/1.9.1/gems/redis-3.0.2/lib/redis.rb:2183:in block in method_missing' /opt/web-collaboration/shared/bundle/ruby/1.9.1/gems/redis-3.0.2/lib/redis.rb:36:inblock in synchronize'
/opt/web-collaboration/shared/bundle/ruby/1.9.1/gems/redis-3.0.2/lib/redis.rb:36:in synchronize' /opt/web-collaboration/shared/bundle/ruby/1.9.1/gems/redis-3.0.2/lib/redis.rb:2182:inmethod_missing'
/opt/web-collaboration/shared/bundle/ruby/1.9.1/gems/redis-sentinel-1.1.2/lib/redis-sentinel/client.rb:41:in discover_master' /opt/web-collaboration/shared/bundle/ruby/1.9.1/gems/redis-sentinel-1.1.2/lib/redis-sentinel/client.rb:15:inconnect_with_sentinel'
/opt/web-collaboration/shared/bundle/ruby/1.9.1/gems/redis-3.0.2/lib/redis/client.rb:282:in ensure_connected' /opt/web-collaboration/shared/bundle/ruby/1.9.1/gems/redis-3.0.2/lib/redis/client.rb:173:inblock in process'
/opt/web-collaboration/shared/bundle/ruby/1.9.1/gems/redis-3.0.2/lib/redis/client.rb:248:in logging' /opt/web-collaboration/shared/bundle/ruby/1.9.1/gems/redis-3.0.2/lib/redis/client.rb:172:inprocess'
/opt/web-collaboration/shared/bundle/ruby/1.9.1/gems/redis-3.0.2/lib/redis/client.rb:84:in call' /opt/web-collaboration/shared/bundle/ruby/1.9.1/gems/redis-3.0.2/lib/redis.rb:378:inblock in exists'
/opt/web-collaboration/shared/bundle/ruby/1.9.1/gems/redis-3.0.2/lib/redis.rb:36:in block in synchronize' /opt/web-collaboration/shared/bundle/ruby/1.9.1/gems/redis-3.0.2/lib/redis.rb:36:insynchronize'
/opt/web-collaboration/shared/bundle/ruby/1.9.1/gems/redis-3.0.2/lib/redis.rb:377:in `exists'

/opt/web-collaboration/shared/bundle/ruby/1.9.1/gems/redis-namespace-1.2.1/lib/redis/namespace.rb:

ERR Wrong number of commands for 'sentinel is-master-down-by-addr' (Redis::CommandError)

Good day.
I would be very grateful for any ideas to solve this problem.
when trying to work with redis in sentinel (3 servers) have problem

ERR Wrong number of commands for 'sentinel is-master-down-by-addr' (Redis::CommandError)

Very simple test code:

require 'redis'
require 'redis-sentinel'

r = Redis.new(master_name: 'mymaster',
    sentinels: [{host:'10.129.0.174',port:6379},{host:'10.129.0.210',port:6379},{host:'10.129.0.249',port:6379}],
    failover_reconnect_timeout: 50,
    failover_reconnect_wait: 0.01)

puts 'set foo'
puts r.set :foo, 1
sleep 1

Error trace:

$ ruby test.rb
set foo
/usr/local/rvm/gems/ruby-1.9.3-p448/gems/redis-3.0.6/lib/redis/client.rb:85:in `call': ERR Wrong number of commands for 'sentinel is-master-down-by-addr' (Redis::CommandError)
    from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/redis-sentinel-1.3.0/lib/redis-sentinel/client.rb:94:in `call_with_readonly_protection'
    from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/redis-3.0.6/lib/redis.rb:2427:in `block in method_missing'
    from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/redis-3.0.6/lib/redis.rb:36:in `block in synchronize'
    from /usr/local/rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
    from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/redis-3.0.6/lib/redis.rb:36:in `synchronize'
    from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/redis-3.0.6/lib/redis.rb:2426:in `method_missing'
    from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/redis-sentinel-1.3.0/lib/redis-sentinel/client.rb:68:in `discover_master'
    from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/redis-sentinel-1.3.0/lib/redis-sentinel/client.rb:25:in `block in connect_with_sentinel'
    from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/redis-sentinel-1.3.0/lib/redis-sentinel/client.rb:43:in `call'
    from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/redis-sentinel-1.3.0/lib/redis-sentinel/client.rb:43:in `auto_retry_with_timeout'
    from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/redis-sentinel-1.3.0/lib/redis-sentinel/client.rb:24:in `connect_with_sentinel'
    from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/redis-3.0.6/lib/redis/client.rb:292:in `ensure_connected'
    from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/redis-3.0.6/lib/redis/client.rb:179:in `block in process'
    from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/redis-3.0.6/lib/redis/client.rb:258:in `logging'
    from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/redis-3.0.6/lib/redis/client.rb:178:in `process'
    from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/redis-3.0.6/lib/redis/client.rb:84:in `call'
    from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/redis-sentinel-1.3.0/lib/redis-sentinel/client.rb:94:in `call_with_readonly_protection'
    from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/redis-3.0.6/lib/redis.rb:675:in `block in set'
    from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/redis-3.0.6/lib/redis.rb:36:in `block in synchronize'
    from /usr/local/rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
    from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/redis-3.0.6/lib/redis.rb:36:in `synchronize'
    from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/redis-3.0.6/lib/redis.rb:671:in `set'
    from test.rb:10:in `<main>'

local gem status:

*** LOCAL GEMS ***

actionmailer (4.0.1, 3.2.15)
actionpack (4.0.1, 3.2.15)
activemodel (4.0.1, 3.2.15)
activerecord (4.0.1, 3.2.15)
activerecord-deprecated_finders (1.0.3)
activeresource (3.2.15)
activesupport (4.0.1, 3.2.15)
arel (4.0.1, 3.0.3)
atomic (1.1.14)
builder (3.1.4, 3.0.4)
bundler (1.3.5)
bundler-unload (1.0.2)
daemon_controller (1.1.7)
erubis (2.7.0)
executable-hooks (1.2.6)
hike (1.2.3)
i18n (0.6.5)
journey (1.0.4)
json (1.8.1)
mail (2.5.4)
mime-types (1.25)
minitest (4.7.5)
multi_json (1.8.2)
passenger (4.0.2)
polyglot (0.3.3)
rack (1.5.2, 1.4.5)
rack-cache (1.2)
rack-ssl (1.3.3)
rack-test (0.6.2)
rails (3.2.15)
railties (4.0.1, 3.2.15)
rake (10.1.0)
rdoc (3.12.2)
redis (3.0.6)
redis-sentinel (1.3.0)
rubygems-bundler (1.4.2)
rvm (1.11.3.8)
sprockets (2.10.0, 2.2.2)
sprockets-rails (2.0.1)
thor (0.18.1)
thread_safe (0.1.3)
tilt (1.4.1)
treetop (1.4.15)
tzinfo (0.3.38)

Servers:
version redis on all servers:

Redis server v=2.8.0 sha=00000000:0 malloc=jemalloc-3.2.0 bits=64 build=f7aae098c2221fd3

server configurations:
Server 1:

daemonize yes
timeout 0
loglevel notice
logfile "/var/log/redis/redis.log"

dir "/var/lib/redis"

pidfile "/var/run/redis/redis.pid"
port 6379
bind 10.129.0.174

maxclients 256

sentinel monitor mymaster 10.129.0.210 6379 2 
sentinel down-after-milliseconds mymaster 20000
sentinel failover-timeout mymaster 900000
#sentinel can-failover mymaster yes
sentinel parallel-syncs mymaster 1
#sentinel is-master-down-by-addr 10.129.0.210 6379

sentinel monitor resque 10.129.0.249 6379 2
sentinel down-after-milliseconds resque 20000
sentinel failover-timeout resque 900000
#sentinel can-failover resque yes
sentinel parallel-syncs resque 5

Server 2:

daemonize yes
timeout 0
loglevel warning
logfile /var/log/redis/redis.log 
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /var/lib/redis/
slave-serve-stale-data yes
slave-read-only yes
appendonly no
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
lua-time-limit 5000

pidfile /var/run/redis/redis.pid
port 6379
bind 10.129.0.210

slowlog-log-slower-than 10000
slowlog-max-len 1024
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60

Server 3:

daemonize yes
timeout 0
loglevel warning
logfile /var/log/redis/redis.log 
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /var/lib/redis/
slave-serve-stale-data yes
slave-read-only yes
appendonly no
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
lua-time-limit 5000

pidfile /var/run/redis/redis.pid
port 6379
bind 10.129.0.249
slaveof 10.129.0.210 6379

slowlog-log-slower-than 10000
slowlog-max-len 1024
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60

Resque: Redis Timeout Error ...

Hi all.
I have used resque with redis-sentinel.
All works OK, except after long unicorn idle time, resque console begin to raise Redis Timeout error.
If I do service unicorn reload, then Resque console opens correctly.
My redis init configuration:

  Resque.redis = Redis.new(master_name: 'redis-master',
                               sentinels: [
                                   {:host => 'server1', :port => 26379},
                                   {:host => 'server2', :port => 26379}
                               ],
                               failover_reconnect_timeout: 30)

my redis-sentinel.conf

sentinel monitor redis-master 172.21.78.138 6379 2
sentinel down-after-milliseconds redis-master 5000
sentinel failover-timeout redis-master 900000
sentinel can-failover redis-master yes
sentinel parallel-syncs redis-master 1

Not possible to set db: ?

Hi,

first of all thanks a lot for your work :).

Everything works fine. I am using redis, redis-sentinel and sidekiq in a Rails application. Now I am stumbling over the problem, that configuring a db: is not possible. The setup is as follows:

sentinels = {
  development:
    [
      { host: "127.0.0.1", port: 6379 }
    ],
  preview:
    [
      { host: "x.x.x.41", port: 26379 },
      { host: "x.x.x.41", port: 26380 },
      { host: "x.x.x.30", port: 26379}
    ]
}

redis_connection = proc {
  Redis.current = Redis.new(master_name: "mymaster", sentinels: sentinels[Rails.env], db: 12)
}
redis = ConnectionPool.new(size: 10, &redis_connection)

Sidekiq.configure_server do |config|
  config.redis = redis
end

Sidekiq.configure_client do |config|
  config.redis = redis
end

This is pretty straight forward but as far as I can see, the db: 12 is not used. When reading the code I can see, that with the given options hash (https://github.com/flyerhzm/redis-sentinel/blob/master/lib/redis-sentinel/client.rb#L9) an initialize_without_sentinel(options) is called and aliased to initialize. So this should be initialize from the Redis gem.

When booting the Rails application and starting the rails console, the db is not shown. It should be sth. like

2.1.1 :001 > r = Redis.current
#<Redis client v3.1.0 for redis://127.0.0.1:6379/12>

When doing this simply in irb, it works.

Any idea where the problem is?

Thanks a lot and cheers

Andy

Wrong setup of sentinels raise exception

Hello,

I just found out my bad config:

production:
  :master_name: master_name
  :sentinels:
  - :host: localhost
    :port: 23340
    :db: 1
  - :host: localhost
    :port: 23341
    :db: 1
  :failover_reconnect_timeout: 30
  :failover_reconnect_wait: 0.0001

When you set :db: for redis sentinel like this, it will raise exception:

Redis::CommandError: ERR unknown command 'select'
    from /Users/bartas/.rvm/gems/ruby-2.0.0-p195@pathfinder-ruby-2-2/gems/redis-3.0.4/lib/redis/client.rb:85:in `call'
    from /Users/bartas/.rvm/gems/ruby-2.0.0-p195@pathfinder-ruby-2-2/gems/redis-3.0.4/lib/redis/client.rb:71:in `connect'
    from /Users/bartas/.rvm/gems/ruby-2.0.0-p195@pathfinder-ruby-2-2/gems/redis-sentinel-1.2.0/lib/redis-sentinel/client.rb:28:in `connect_with_sentinel'
    from /Users/bartas/.rvm/gems/ruby-2.0.0-p195@pathfinder-ruby-2-2/gems/redis-3.0.4/lib/redis/client.rb:290:in `ensure_connected'
    from /Users/bartas/.rvm/gems/ruby-2.0.0-p195@pathfinder-ruby-2-2/gems/redis-3.0.4/lib/redis/client.rb:177:in `block in process'
    from /Users/bartas/.rvm/gems/ruby-2.0.0-p195@pathfinder-ruby-2-2/gems/redis-3.0.4/lib/redis/client.rb:256:in `logging'
    from /Users/bartas/.rvm/gems/ruby-2.0.0-p195@pathfinder-ruby-2-2/gems/redis-3.0.4/lib/redis/client.rb:176:in `process'
    from /Users/bartas/.rvm/gems/ruby-2.0.0-p195@pathfinder-ruby-2-2/gems/redis-3.0.4/lib/redis/client.rb:84:in `call'
    from /Users/bartas/.rvm/gems/ruby-2.0.0-p195@pathfinder-ruby-2-2/gems/redis-3.0.4/lib/redis.rb:2240:in `block in method_missing'
    from /Users/bartas/.rvm/gems/ruby-2.0.0-p195@pathfinder-ruby-2-2/gems/redis-3.0.4/lib/redis.rb:36:in `block in synchronize'

This is becase it tries to connect to sentinel with database 1. I am not sure that sentinel can run in another database than 0.

What do you think about this?

Incorrect master name causes endless flood of connection attempts

In the event that redis-sentinel is misconfigured with valid sentinels but an invalid master name, the logic in discover_slaves will endlessly loop through all configured sentinels with no timeout.

This has caused us some grief, because it's easy to run out of file descriptors with TIME_WAIT connections to Redis.

I'm happy to work on a PR for this, but wanted to touch base first to see if anyone had preferences on how to address this. I'm leaning toward tracking whether we have connected to every sentinel and waiting for some amount of time after failing on every sentinel (up to failover_reconnect_timeout or perhaps a new configurable timout).

Thanks.

Work with Redis sentinel 2.8.X

Hi.
Do you test redis-sentinel gem with redis 2.8 ?

Use with Rails.config.cache_store?

Is it possible to use this gem with Rails.config.cache_store? I'm setting the Rails cache_store like so:

config.cache_store = :redis_store, { master_name: 'master', sentinels: [{host: 'localhost', port: 26379}], :expires_in => 1.hour }

But accessing the Rails cache gives a Redis::CannotConnectError caused by "try_next_sentinel" method. Upon further investigation, it looks like Rails.cache is a RedisStore object with an empty @options[:sentinels] array, even though its @options[:master_name] variable is set correctly.

Load balancing between redis slaves

Hello,

I am working on redis backup and failover situation where I have 3 frontend server, every server has its own redis + fronted application.

1 redis server will be started as master, other as slaves.

Redis Sentinel knows all servers in cluster and it would be very nice to use connection for read commands to nearest (fastest ping) redis server (every server will ask its own redis server for read and write into only one master)

When one of slaves goes down, then application using this server will switch into another redis server (doesn't matter if it is slave or master)

When master goes down, all read commands will proceed without fail (slaves will be used), write command will be with errors till redis Sentinel will switch one of slaves into master mode. Then all write commands will be switched to this new master.

Currently all requests are going just into master (if I read code of redis-sentinel right) and all slaves are not in use :(

What do you think about this approach?

If you like it, I will try to extend redis-sentinel to use this scenario.

Ondrej Bartas

Logging to syslog

Does sentinel support logging to syslog like redis does?

Currently we have:

loglevel verbose
logfile "/var/log/redis/sentinel.log"