Skip to content

New error message handling is broken when using sentinels #182

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jlledom opened this issue Mar 21, 2024 · 10 comments · Fixed by #183
Closed

New error message handling is broken when using sentinels #182

jlledom opened this issue Mar 21, 2024 · 10 comments · Fixed by #183

Comments

@jlledom
Copy link
Contributor

jlledom commented Mar 21, 2024

Release 0.21.0 introduced a new error message handling: #178

However, this blocks the execution when trying to connect to an invalid sentinel client. Check this example:

# frozen_string_literal: true

require "bundler/inline"

gemfile(true) do
  source "https://rubygems.org"

  gem 'redis-client'#, '0.20.0'
end

def main
  redis_config = RedisClient.sentinel(
    name: "mymaster",
    sentinels: [
      { host: "127.0.0.1", port: 26380 },
      { host: "127.0.0.1", port: 26381 },
    ],
    role: :master
  )
  redis = redis_config.new_pool(timeout: 0.5, size: 5)

  while true
    begin
      sleep 1
      result = redis.call("PING") # => "PONG"
      puts result
    rescue StandardError => e
      puts e.message
      retry
    end
  end
end

main

When using 0.20.0. This is the result:

$ ruby redis-client-sentinels-inavlid.rb 
Fetching gem metadata from https://rubygems.org/..
Resolving dependencies...
Using bundler 2.2.25
Using connection_pool 2.4.1
Using redis-client 0.20.0
No sentinels available
No sentinels available
No sentinels available
No sentinels available
...

And this is the result using 0.21.0:

$ ruby redis-client-sentinels-inavlid.rb 
Fetching gem metadata from https://rubygems.org/..
Resolving dependencies...
Using bundler 2.2.25
Using connection_pool 2.4.1
Using redis-client 0.21.0
Traceback (most recent call last):

After taking a quick look at the code, I'd say this happens because the new HasConfig#message method calls to config.server_url:

def message
return super unless config
"#{super} (#{config.server_url})"
end

Which calls to host:

def server_url
if path
url = "unix://#{path}"
if db != 0
url = "#{url}?db=#{db}"
end
else
url = "redis#{'s' if ssl?}://#{host}:#{port}"
if db != 0
url = "#{url}/#{db}"
end
end
url
end

Which, if I'm not wrong, ends up calling to resolve_master, and trying to connect again, thus raising the error again.

def resolve_master
each_sentinel do |sentinel_client|
host, port = sentinel_client.call("SENTINEL", "get-master-addr-by-name", @name)
next unless host && port
refresh_sentinels(sentinel_client)
return Config.new(host: host, port: Integer(port), **@client_config)
end
rescue ConnectionError
raise ConnectionError, "No sentinels available"
else
raise ConnectionError, "Couldn't locate a replica for role: #{@name}"
end

@byroot
Copy link
Member

byroot commented Mar 21, 2024

FYI @stanhu

@stanhu
Copy link
Contributor

stanhu commented Mar 21, 2024

Is the simplest solution here to catch ConnectionError in message and give up?

@stanhu
Copy link
Contributor

stanhu commented Mar 21, 2024

Alternatively, we could use a different method or use an argument to config that avoids the call to resolve_master or resolve_replica.

@jlledom
Copy link
Contributor Author

jlledom commented Mar 21, 2024

@stanhu IMO catching errors in message is the way to go. Because showing the URL in the message is not a vital feature. If we can have it, cool, but if something fails it's ok to just return super, not a big deal. I would catch any error, not only ConnectionError

@casperisfine
Copy link
Collaborator

I don't like the rescue idea, I think we should just not call anything that could fail from here.

So we probably need a version of server_url that just return something else if it wasn't resolved for a sentinel client.

@casperisfine
Copy link
Collaborator

@stanhu if you want to fix it it's all good, but if you don't have time let me know and I'll find the time.

@stanhu
Copy link
Contributor

stanhu commented Mar 21, 2024

@casperisfine I'll fix this.

@stanhu
Copy link
Contributor

stanhu commented Mar 21, 2024

So we probably need a version of server_url that just return something else if it wasn't resolved for a sentinel client.

The issue is that config calls resolve_master or resolve_replica, so we need some config object that doesn't attempt that.

stanhu added a commit to stanhu/redis-client that referenced this issue Mar 21, 2024
redis-rb#178 introduced
a regression that caused a ConnectionError to be thrown to the
caller if the Sentinel master or replica could not be resolved.

When a ConnectionError is thrown, the error message handler
would attempt to retrieve `config.server_url`, but this in turn
causes another Sentinel resolution to be attempted.

We avoid this by adding a `resolved?` method that will indicate
whether the config can be used. The error handler won't attempt
to provide more details if the config has yet to be resolved.

Closes redis-rb#182
stanhu added a commit to stanhu/redis-client that referenced this issue Mar 21, 2024
redis-rb#178 introduced
a regression that caused a ConnectionError to be thrown to the
caller if the Sentinel master or replica could not be resolved.

When a ConnectionError is thrown, the error message handler
would attempt to retrieve `config.server_url`, but this in turn
causes another Sentinel resolution to be attempted.

We avoid this by adding a `resolved?` method that will indicate
whether the config can be used. The error handler won't attempt
to provide more details if the config has yet to be resolved.

Closes redis-rb#182
stanhu added a commit to stanhu/redis-client that referenced this issue Mar 21, 2024
redis-rb#178 introduced
a regression that caused a `ConnectionError` to be thrown to the
caller if the Sentinel master or replica could not be resolved.

When a `ConnectionError` is thrown, the error message handler would
attempt to retrieve `config.server_url`, but this in turn causes
another Sentinel resolution to be attempted.

We avoid this by adding a `resolved?` method that will indicate
whether the config can be used. The error handler won't attempt
to provide more details if the config has yet to be resolved.

Closes redis-rb#182
@stanhu
Copy link
Contributor

stanhu commented Mar 21, 2024

I think #183 should solve this.

@stanhu
Copy link
Contributor

stanhu commented Mar 22, 2024

It looks like this was fixed in v0.21.1. Thanks for the quick review and release!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants