Skip to content

connection pool session keepalive #483

Closed

Description

@azrle

TL;DR

  1. Connections in connection pool should response to keepalive message (probably via Net::SSH::Connection::Session.loop);
  2. Connection pool may need to detect closed session and/or recover from it.

Details

We are using capistrano to perform deployments and we noticed that the following errors occurred sometimes.

17:38:38 cap aborted!
17:38:38 SSHKit::Runner::ExecuteError: Exception while executing as test@test-server: closed stream
17:38:38 
17:38:38 
17:38:38 Caused by:
17:38:38 IOError: closed stream

After some investigations, we notice that it may because the SSHKit doesn't response to keepalive message from ssh server.
Then the ssh server close the stream but the SSHKit still wants to use that connection and writes data to it.

The following codes are tested on Ubuntu 18.04 with SSHKit 1.21.0 (net-ssh 6.1.0) and following SSH config:

  • server (/etc/ssh/sshd_config)
ClientAliveInterval 10
ClientAliveCountMax 3
  • client (/etc/ssh/ssh_config)
ServerAliveCountMax  3
ServerAliveInterval 15
  • run on client
require 'sshkit'
require 'sshkit/dsl'
include SSHKit::DSL

SSHKit::Backend::ConnectionPool::Cache.module_eval do
  def event_loop
    connections.synchronize do
      connections.map(&:last).each(&:loop)
    end
  end
end

SSHKit::Backend::ConnectionPool.module_eval do
  def run_event_loop
    loop do
      caches.values.each(&:event_loop)
      sleep 1
    end
  end
end

# Thread.new { SSHKit::Backend::Netssh.pool.run_event_loop }
SSHKit::Backend::Netssh.pool.idle_timeout = 60
SSHKit.config.output_verbosity = Logger::DEBUG

remote_host = SSHKit::Host.new('test-server')

on remote_host do
  execute :echo, 'ok'
end

p 'sleeping...'
sleep 31
p 'waked up'

on remote_host do
  execute :echo, 'ok'
end
  • Result
  INFO [887c3c9c] Running /usr/bin/env echo ok on test-server
 DEBUG [887c3c9c] Command: /usr/bin/env echo ok
 DEBUG [887c3c9c]       ok
  INFO [887c3c9c] Finished in 0.236 seconds with exit status 0 (successful).
"sleeping..."
"waked up"
  INFO [73d9af55] Running /usr/bin/env echo ok on test-server
 DEBUG [73d9af55] Command: /usr/bin/env echo ok
#<Thread:0x000055d82f3e1400@/path/to/.rbenv/versions/2.5.8/lib/ruby/gems/2.5.0/gems/sshkit-1.21.0/lib/sshkit/runners/parallel.rb:10 run> terminated with exception (report_on_exception is true):
Traceback (most recent call last):
        1: from /path/to/.rbenv/versions/2.5.8/lib/ruby/gems/2.5.0/gems/sshkit-1.21.0/lib/sshkit/runners/parallel.rb:11:in `block (2 levels) in execute'
/path/to/.rbenv/versions/2.5.8/lib/ruby/gems/2.5.0/gems/sshkit-1.21.0/lib/sshkit/runners/parallel.rb:15:in `rescue in block (2 levels) in execute': Exception while executing on host test-server: closed stream (SSHKit::Runner::ExecuteError)
Traceback (most recent call last):
        1: from /path/to/.rbenv/versions/2.5.8/lib/ruby/gems/2.5.0/gems/sshkit-1.21.0/lib/sshkit/runners/parallel.rb:11:in `block (2 levels) in execute'
/path/to/.rbenv/versions/2.5.8/lib/ruby/gems/2.5.0/gems/sshkit-1.21.0/lib/sshkit/runners/parallel.rb:15:in `rescue in block (2 levels) in execute': Exception while executing on host test-server: closed stream (SSHKit::Runner::ExecuteError)

Workaround

  1. Disable or increase ClientAliveCountMax. However, it's not always possible to do that since it can affect other applications and sometimes we don't have permission to do so.
  2. Disable connection pool by set SSHKit::Backend::Netssh.pool.idle_timeout = 0 or less than ClientAliveCountMax*ClientAliveInterval. But it hurts performance.
  3. Run Net::SSH::Connection::Session.loop every second with monkey patch by enabling this line Thread.new { SSHKit::Backend::Netssh.pool.run_event_loop }. Not sure that it's safe to do so and doesn't handle exceptions here. Hence, consider it as a workaround.

Activity

  1. leehambley commented on Aug 19, 2020

    @leehambley
    Member

    Hi @azrle thanks for the message. Do you know if Net::SSH has the same problem directly? It's been a number of years (about 7?) since I wrote the "event" handling code on the interface between SSHKit and Net::SSH, but I don't recall at that time that there was a server HUP message we were failing to respond to. Maybe Net::SSH has added this feature in the meantime, and we missed it, or it was always there and we overlooked it?

    If you can confirm that this is an SSHKit bug, and that the underlying SSH handling library behaves correctly and we're mis-handling, or failing to handle events, I'd be happy to work with you to come up with a patch.

    Thanks!

  2. azrle commented on Aug 20, 2020

    @azrle
    ContributorAuthor

    Thanks for following up!

    As far as I know, it's not a new feature of Net::SSH. Net::SSH will response a message to global request containing keepalive message from ssh server (sshd). However, it can handle the global request only in its event loop. In the case the session is idle, it seems that Net::SSH doesn't automatically trigger the event loop from old versions till now.

    I tried some old versions of Net::SSH with following codes, which may explain it.

    require 'net/ssh'
    
    Net::SSH.start('localhost', nil, keepalive: false, verbose: :debug) do |ssh|
      ssh.exec 'echo ok'
      for i in 0..40
        # ssh.process(0)
        # ssh.loop
        sleep 1
      end
      ssh.exec 'echo ok'
      ssh.loop
    end

    Note that I explicitly disable keepalive messages from Net::SSH (keepalive: false) as it's irrelevant and can be confused with keepalive messages from sshd.

    The sshd config is the same, which means if a non-active client doesn't reply to keepalive message from sshd in about 30 seconds, sshd will close the connection.

    ClientAliveInterval 10
    ClientAliveCountMax 3
    

    We can observe that there no messages exchanged during the sleep.
    And finally we got:

    lib/ruby/gems/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/transport/packet_stream.rb:89:in `next_packet': connection closed by remote host (Net::SSH::Disconnect)
    

    If we enable ssh.loop (for net-ssh >= 4.2.0, actually net-ssh/net-ssh#531) or ssh.process(0), we can see that now Net::SSH is replying keepalive messages from sshd and the second echo succeeded.

    D, [2020-08-20T03:50:50.282152 #18363] DEBUG -- socket[91af3c]: read 68 bytes
    D, [2020-08-20T03:50:50.282304 #18363] DEBUG -- socket[91af3c]: received packet nr 16 type 80 len 44
    I, [2020-08-20T03:50:50.282380 #18363]  INFO -- net.ssh.connection.session[8de118]: global request received: keepalive@openssh.com true
    D, [2020-08-20T03:50:50.282474 #18363] DEBUG -- socket[91af3c]: queueing packet nr 14 type 82 len 28
    D, [2020-08-20T03:50:50.282568 #18363] DEBUG -- socket[91af3c]: sent 52 bytes
    

    Hence, IMHO, sshkit should trigger Net::SSH event loop somehow.
    And it might be good to rescue from the exception that the connection is closed.
    (Difficult to tell it's safe to retry the command already sent to the remote? But in keepalive case when the session is idle, it can be safe to mark connection dead.)

  3. azrle commented on Sep 11, 2020

    @azrle
    ContributorAuthor

    @leehambley It has been a while. What do you think about this issue?

  4. leehambley commented on Sep 11, 2020

    @leehambley
    Member

    I think it's very worthwhile, but I'm not able to work on a patch of this complexity. If you prepare a patch, I will do everything to support you so we can get it merged into SSHKit and push a new release.

  5. azrle commented on Sep 11, 2020

    @azrle
    ContributorAuthor

    I see. Let me try to fix this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions