Repository navigation
Closed
Description
TL;DR
- Connections in connection pool should response to keepalive message (probably via
Net::SSH::Connection::Session.loop); - Connection pool may need to detect closed session and/or recover from it.
Details
We are using capistrano to perform deployments and we noticed that the following errors occurred sometimes.
17:38:38 cap aborted!
17:38:38 SSHKit::Runner::ExecuteError: Exception while executing as test@test-server: closed stream
17:38:38
17:38:38
17:38:38 Caused by:
17:38:38 IOError: closed stream
After some investigations, we notice that it may because the SSHKit doesn't response to keepalive message from ssh server.
Then the ssh server close the stream but the SSHKit still wants to use that connection and writes data to it.
The following codes are tested on Ubuntu 18.04 with SSHKit 1.21.0 (net-ssh 6.1.0) and following SSH config:
- server (
/etc/ssh/sshd_config)
ClientAliveInterval 10
ClientAliveCountMax 3
- client (
/etc/ssh/ssh_config)
ServerAliveCountMax 3
ServerAliveInterval 15
- run on client
require 'sshkit'
require 'sshkit/dsl'
include SSHKit::DSL
SSHKit::Backend::ConnectionPool::Cache.module_eval do
def event_loop
connections.synchronize do
connections.map(&:last).each(&:loop)
end
end
end
SSHKit::Backend::ConnectionPool.module_eval do
def run_event_loop
loop do
caches.values.each(&:event_loop)
sleep 1
end
end
end
# Thread.new { SSHKit::Backend::Netssh.pool.run_event_loop }
SSHKit::Backend::Netssh.pool.idle_timeout = 60
SSHKit.config.output_verbosity = Logger::DEBUG
remote_host = SSHKit::Host.new('test-server')
on remote_host do
execute :echo, 'ok'
end
p 'sleeping...'
sleep 31
p 'waked up'
on remote_host do
execute :echo, 'ok'
end- Result
INFO [887c3c9c] Running /usr/bin/env echo ok on test-server
DEBUG [887c3c9c] Command: /usr/bin/env echo ok
DEBUG [887c3c9c] ok
INFO [887c3c9c] Finished in 0.236 seconds with exit status 0 (successful).
"sleeping..."
"waked up"
INFO [73d9af55] Running /usr/bin/env echo ok on test-server
DEBUG [73d9af55] Command: /usr/bin/env echo ok
#<Thread:0x000055d82f3e1400@/path/to/.rbenv/versions/2.5.8/lib/ruby/gems/2.5.0/gems/sshkit-1.21.0/lib/sshkit/runners/parallel.rb:10 run> terminated with exception (report_on_exception is true):
Traceback (most recent call last):
1: from /path/to/.rbenv/versions/2.5.8/lib/ruby/gems/2.5.0/gems/sshkit-1.21.0/lib/sshkit/runners/parallel.rb:11:in `block (2 levels) in execute'
/path/to/.rbenv/versions/2.5.8/lib/ruby/gems/2.5.0/gems/sshkit-1.21.0/lib/sshkit/runners/parallel.rb:15:in `rescue in block (2 levels) in execute': Exception while executing on host test-server: closed stream (SSHKit::Runner::ExecuteError)
Traceback (most recent call last):
1: from /path/to/.rbenv/versions/2.5.8/lib/ruby/gems/2.5.0/gems/sshkit-1.21.0/lib/sshkit/runners/parallel.rb:11:in `block (2 levels) in execute'
/path/to/.rbenv/versions/2.5.8/lib/ruby/gems/2.5.0/gems/sshkit-1.21.0/lib/sshkit/runners/parallel.rb:15:in `rescue in block (2 levels) in execute': Exception while executing on host test-server: closed stream (SSHKit::Runner::ExecuteError)
Workaround
- Disable or increase
ClientAliveCountMax. However, it's not always possible to do that since it can affect other applications and sometimes we don't have permission to do so. - Disable connection pool by set
SSHKit::Backend::Netssh.pool.idle_timeout = 0or less thanClientAliveCountMax*ClientAliveInterval. But it hurts performance. - Run
Net::SSH::Connection::Session.loopevery second with monkey patch by enabling this lineThread.new { SSHKit::Backend::Netssh.pool.run_event_loop }. Not sure that it's safe to do so and doesn't handle exceptions here. Hence, consider it as a workaround.
Reacted by keitarou
Metadata
Metadata
Assignees
Labels
No labels
Activity
leehambley commentedon Aug 19, 2020
Hi @azrle thanks for the message. Do you know if Net::SSH has the same problem directly? It's been a number of years (about 7?) since I wrote the "event" handling code on the interface between SSHKit and Net::SSH, but I don't recall at that time that there was a server HUP message we were failing to respond to. Maybe Net::SSH has added this feature in the meantime, and we missed it, or it was always there and we overlooked it?
If you can confirm that this is an SSHKit bug, and that the underlying SSH handling library behaves correctly and we're mis-handling, or failing to handle events, I'd be happy to work with you to come up with a patch.
Thanks!
azrle commentedon Aug 20, 2020
Thanks for following up!
As far as I know, it's not a new feature of Net::SSH. Net::SSH will response a message to global request containing keepalive message from ssh server (sshd). However, it can handle the global request only in its event loop. In the case the session is idle, it seems that Net::SSH doesn't automatically trigger the event loop from old versions till now.
I tried some old versions of Net::SSH with following codes, which may explain it.
Note that I explicitly disable keepalive messages from Net::SSH (
keepalive: false) as it's irrelevant and can be confused with keepalive messages fromsshd.The sshd config is the same, which means if a non-active client doesn't reply to keepalive message from
sshdin about 30 seconds,sshdwill close the connection.We can observe that there no messages exchanged during the sleep.
And finally we got:
If we enable
ssh.loop(for net-ssh >= 4.2.0, actually net-ssh/net-ssh#531) orssh.process(0), we can see that now Net::SSH is replying keepalive messages from sshd and the secondechosucceeded.Hence, IMHO, sshkit should trigger Net::SSH event loop somehow.
And it might be good to rescue from the exception that the connection is closed.
(Difficult to tell it's safe to retry the command already sent to the remote? But in keepalive case when the session is idle, it can be safe to mark connection dead.)
azrle commentedon Sep 11, 2020
@leehambley It has been a while. What do you think about this issue?
leehambley commentedon Sep 11, 2020
I think it's very worthwhile, but I'm not able to work on a patch of this complexity. If you prepare a patch, I will do everything to support you so we can get it merged into SSHKit and push a new release.
azrle commentedon Sep 11, 2020
I see. Let me try to fix this.