Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SSH Connection Closed While Looping over a command list #689

Open
mountaintop1 opened this issue Sep 16, 2024 · 14 comments
Open

SSH Connection Closed While Looping over a command list #689

mountaintop1 opened this issue Sep 16, 2024 · 14 comments

Comments

@mountaintop1
Copy link

With respect to below, ssh connection closes after the first command. This works with Juniper device but failed with Cisco devices.

Am I missing something? It is looking like there is an exit after the first command.

async def get_show_using_asynssh(dict_data: dict):
async with asyncssh.connect(host=dict_data["ip_address"], username=dict_data["username"], password=dict_data["password"],
known_hosts=None) as conn:
output = {}
commands = ['show mac address-table', 'show lldp neighbors detail', 'show vlan']
try:
for cmd in commands: # this is a multiple command coroutine
out = (await conn.run(cmd, term_size=(5000000000, 2400000), check=True)).stdout
if out:
output[cmd.split()[1]] = out
else:
output[cmd.split()[1]] = ''
print('Finished send command string.............')

    except Exception as e:
        print(e)
        return f"Error: {str(e)}"
@ronf
Copy link
Owner

ronf commented Sep 17, 2024

The code you posted got mangled, so it's hard to see what the indentation was. However, assuming that had the calls to conn.run() properly inside the async with asyncssh.connect(), you're probably running into an issue that many embedded SSH servers only support a single session per connection. An attempt to start additional connections will fail on such servers.

Your options are to either do a separate asyncssh.connect() for each command you want to send, which means paying the cost of authenticating each time, or you need to open an interactive shell and send commands one after another on a single session. This is tricky to do, though, as many of these servers only accept input when they are done running each command, and unless you know the expected output of the command, it's hard to know when to stop reading one command's output and when to start sending the next command.

If you look through the closed issues here, you'll see a number of examples about different ways of handling this. One example is #121. Another example is #227, and there are probably others as well as this topic comes up repeatedly. Unfortunately, there's no simple answer here. It really depends on the device and your ability to get the device to output something unique enough at the end of each command's output to give you a reliable indicator of when it is safe to send the next command. For instance, if you can configure a unique prompt on the device which won't show up in the command output, you can do a readuntil() on that.

@mountaintop1
Copy link
Author

I tried the #121 below it didnt work. The device is a Cisco IOS XE. I need your guidance please. My plan is to capture the output of each command in the list and save as a key to a dictionary. example is
output = {"mac": "", "vlan": "", "lldp": ""}

`async def get_show_using_asynssh(dict_data: dict):

async with asyncssh.connect(host=dict_data["ip_address"], username=dict_data["username"], 
                                                      password=dict_data["password"], known_hosts=None) as conn:
    output = {}
    
    try:
	commands =  ['show mac address-table', 'show lldp neighbors detail', 'show vlan']
	process = await conn.create_process(command='<remote process, eg: bc>')
		
	await asyncio.wait_for(process.stdout.readuntil(PROMPT), timeout=10)
	for cmd in commands:
		process.stdin.write(cmd + '\n')
		result = ''
		try:
			result+=await\
					asyncio.wait_for(
						process.stdout.readuntil(PROMPT),timeout=10)
		except Exception as err:
			logging.warning("prompt timeout step {}".format(err))
		print(result, end='')
    except Exception as e:
        return f"Error: {str(e)}"

Below is the error I got.`

1941 bytes read on a total of undefined expected bytes
1935 bytes read on a total of undefined expected bytes
1909 bytes read on a total of undefined expected bytes
1935 bytes read on a total of undefined expected bytes
1935 bytes read on a total of undefined expected bytes
1942 bytes read on a total of undefined expected bytes

Thanks for your help!

@ronf
Copy link
Owner

ronf commented Sep 17, 2024

The error you're seeing generally indicates that you received an end-of-file from the remote system before getting the prompt you told it to look for. I don't see the definition of PROMPT in the sample code above, but make sure that the remote system actually does output a prompt, and that you set PROMPT to match whatever it prints when command output ends and it is ready to receive a new command.

Also, for this to work, you shouldn't be providing a command. You need it to open an interactive shell, which occurs when you call create_process() with no command set.

@mountaintop1
Copy link
Author

It seems there is no straight forward solution for this just like we use Netmiko or Paramiko for Cisco device. Asynssh works with Juniper but this issue with Cisco is major one. It will be good if you have a solution for this. I have edited the code. I now get some data from the command output but not all. I dont the the output of show vlan command. Am I missing anything here?

PROMPT = r'#'

async def get_show_using_asynssh(dict_data: dict):
    async with asyncssh.connect(host=dict_data["ip_address"], username=dict_data["username"], 
                                                          password=dict_data["password"], known_hosts=None) as conn:
        output = {}
        
        try:
                commands =  ['show mac address-table', 'show lldp neighbors detail',  'show vlan']
                process = await conn.create_process()
                process.stdin.write('terminal lenght 0' + '\n')
                
                await asyncio.wait_for(process.stdout.readuntil(PROMPT), timeout=30)
                for cmd in commands:
                    try:
                        process.stdin.write(cmd + '\n')
                        output[cmd.split()[1]] await asyncio.wait_for(process.stdout.readuntil(PROMPT), timeout=30)
                    except Exception as err:
                        logging.warning("prompt timeout step {}".format(err))
        except Exception as e:
            return f"Error: {str(e)}"

@ronf
Copy link
Owner

ronf commented Sep 17, 2024

By any chance, does any of the output in any of the commands contain '#' in the output? If so, your prompt isn't unique enough to work reliably.

I also see what looks like a typo in the "terminal length 0" command string. That might prevent you from getting to a prompt if the amount of output is larger than the default terminal length.

@mountaintop1
Copy link
Author

Below is the prompt after login and after command output.

cisco_switch_prompt#

'terminal length 0' in cisco allows you to display ouput without breaks

below is an example of command and output;

device_name#sh vlan

VLAN Name                             Status    Ports
---- -------------------------------- --------- -------------------------------
1    default                          active    Te1/1/3, Te1/1/4, Ap1/0/1
60   MGMT                             active
1002 fddi-default                     act/unsup
1003 token-ring-default               act/unsup
1004 fddinet-default                  act/unsup
1005 trnet-default                    act/unsup
1181 1181_41ra0551                    active    Gi1/0/1, Gi1/0/2, Gi1/0/3, Gi1/0/4, Gi1/0/5, Gi1/0/6
                                                Gi1/0/7, Gi1/0/8, Gi1/0/9, Gi1/0/10, Gi1/0/11, Gi1/0/12
                                                Gi1/0/13, Gi1/0/14, Gi1/0/15, Gi1/0/16, Te1/0/17, Te1/0/18
                                                Te1/0/19, Te1/0/20, Te1/0/21, Te1/0/22, Te1/0/23
1394 1394_41ra0551                    active    Te1/0/24
2181 2181_41ra0551                    active
3181 3181_41ra0551                    active
4001 blackhole                        active

VLAN Type  SAID       MTU   Parent RingNo BridgeNo Stp  BrdgMode Trans1 Trans2
---- ----- ---------- ----- ------ ------ -------- ---- -------- ------ ------
1    enet  100001     1500  -      -      -        -    -        0      0
60   enet  100060     1500  -      -      -        -    -        0      0
1002 fddi  101002     1500  -      -      -        -    -        0      0
1003 tr    101003     1500  -      -      -        -    -        0      0
1004 fdnet 101004     1500  -      -      -        ieee -        0      0
1005 trnet 101005     1500  -      -      -        ibm  -        0      0
1181 enet  101181     1500  -      -      -        -    -        0      0
1394 enet  101394     1500  -      -      -        -    -        0      0
2181 enet  102181     1500  -      -      -        -    -        0      0
3181 enet  103181     1500  -      -      -        -    -        0      0
4001 enet  104001     1500  -      -      -        -    -        0      0

Remote SPAN VLANs
------------------------------------------------------------------------------


Primary Secondary Type              Ports
------- --------- ----------------- ------------------------------------------

device_name#

thanks for your time

@ronf
Copy link
Owner

ronf commented Sep 18, 2024

'terminal length 0' in cisco allows you to display ouput without breaks

Yes, I understand. My point was that your sample code included 'terminal lenght 0' instead of 'terminal length 0'. This would cause it to try and paginate the output, probably causing it to stop before it got back to the prompt. This would make the readuntil() time out.

I would also set PROMPT to something more than just '#', unless you're sure that none of the commands you were running would ever include a '#' in their output.

@mountaintop1
Copy link
Author

Sorry I didnt get you here
"My point was that your sample code included 'terminal lenght 0' instead of 'terminal length 0'"

@mountaintop1
Copy link
Author

By the way I see this as part of the output

'device_name#') WARNING:root:prompt timeout step WARNING:root:prompt timeout step WARNING:root:prompt timeout step WARNING:root:prompt timeout step WARNING:root:prompt timeout step ('show mac address-table\r\n' ' Mac Address Table\r\n' '-------------------------------------------\r\n' '\r\n'

@ronf
Copy link
Owner

ronf commented Sep 18, 2024

Take a look at your sample code above. One of the lines is:

                process.stdin.write('terminal lenght 0' + '\n')

This appears to be a typo in the command you are trying to run to disable pagination.

@mountaintop1
Copy link
Author

Oh I see it now ht vs th........ Thanks so much I got it to work now by adding 'terminal length 0' to the list of command

async def get_show_using_asynssh(dict_data: dict):
    PROMPT = r'#'
    async with asyncssh.connect(host=dict_data["ip_address"], username=dict_data["username"] 
                          password=dict_data["password"], known_hosts=None,) as conn:
        output = {}
        commands =  ['terminal length 0', 'show mac address-table', 'show lldp neighbors detail', 'sh vlan brief']
         process = await conn.create_process()
                              
          await asyncio.wait_for(process.stdout.readuntil(PROMPT), timeout=30)
           for cmd in commands:
                    try:
                        process.stdin.write(cmd + '\n')
                        output[cmd.split()[1]] = await asyncio.wait_for(process.stdout.readuntil(PROMPT), timeout=20)
                    except Exception as err:
                        logging.warning("prompt timeout step {}".format(err))

Thanks for helping me see this through.
By the way, I am testing the speed of Asyncio (asynssh) vs Concurrent futures (Netmiko) for collecting data and configuring both cisco and Juniper switches.

I performed the operation on 150 switches and my result is
Concurrent Future took about 75 secs
Asynssh took 18 secs

Asynssh is way better and faster. Thanks so much.

@ronf
Copy link
Owner

ronf commented Sep 18, 2024

Glad to hear you got it working!

Also, thanks for sharing the results of your performance tests... That's a pretty nice improvement!

Opening many connections at once is definitely one of AsyncSSH's strengths, as it doesn't require separate processes (or threads per connection (or even per session) the way most SSH client libraries do. If you are running on a multi-core client, you can improve performance even more by running multiple event loops at once, but you only want a small number of threads (one per available core) in that case, rather than opening a separate process or thread for each connection.

@mountaintop1
Copy link
Author

How can I run multiple event loops at once? example

Thanks

@ronf
Copy link
Owner

ronf commented Sep 18, 2024

I don't have an example handy, and it can get pretty involved depending on how independent your tasks are, but it basically involves using the "multiprocessing" library to create a pool of processes, letting each process create its own asyncio event loop, and then dividing up the work you are doing so that is gets spread across the multiple threads.

For example, let's say that you knew up front what the 150 servers were you wanted to pull data from, and you wanted to spread the work across 5 cores. You could simply break the list of servers into 5 lists of 30 servers each, and then you could create 5 separate processes and feed each process one of the 5 lists. In your main process, you'd call join() on each of the 5 processes, collecting the results from each until you had all 150 responses.

Fancier versions of this could involve having a multiprocess queue, where you have your main task add work items to the queue and let the child processes pull work off the queue, writing the results back to another multiprocess queue, where you limit the number of asyncio tasks each process starts pulling work from the pool until it hits some limit on the number of concurrent tasks you want it to start. After that, each task that completes would trigger it to get another one until the queue was exhausted.

I don't have any direct experience with it, but you might want to experiment with https://aiomultiprocess.omnilib.dev/en/stable/guide.html to do some of the work for you, providing an easy way to schedule async tasks across multiple processes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants