SSH Collection Module

The SSH Collection Module provides data collection for Linux and UNIX-class systems as part of the RISC Networks engagement process. It uses the Secure Shell (SSH) protocol to communicate with in-scope discovered devices to collect identifying inventory data as well as ongoing performance data.

SSH is an industry-standard protocol that provides an encrypted, authenticated channel of communication between devices, forming the backbone of most systems’ orchestration frameworks today. The SSH Collection Module acts as an SSH client to communicate with SSH servers running on devices in the environment. The SSH Collection Module utilizes OpenSSH, the current de-facto standard SSH distribution.

Previously, collection from Linux and UNIX-class systems was conducted using the SNMP protocol. While SNMP is still supported for these device types, the SSH Collection Module can replace the SNMP protocol for all data collection. The SSH Collection Module does not support collection from network devices, which still require the use of SNMP. Details on system eligibility for participation in the SSH Collection Module are provided below.

Please note that the SSH Collection Module is a separate feature from the CLI data collection process for Cisco network devices using the telnet/SSH protocols.

On this page:

Overview

When the SSH credentials section has been opted-in on the RN150 Virtual Appliance, and at least one SSH credential has been entered, the SSH Collection Module will be utilized to attempt to collect data from discovered devices that meet the eligibility requirements.

The RN150 Virtual Appliance acts as an SSH client that will connect to devices, authenticate using the provided credentials, issue commands on the remote system, and collect the output of those commands. The commands that are issued using the SSH Collection Module are documented below.


Supported Operating Systems

The SSH Collection Module does not currently support all UNIX-class operating systems, although additional systems will be released over time. The currently supported systems are:

  • Linux
  • IBM AIX

For Linux, the SSH Collection Module is not distribution-specific, but due to the wide variety of distributions and subtle differences between them, it may be the case that niche or very old distributions are not accounted for by the SSH Collection Module. In this case, a support case can be opened for evaluation, and/or SNMP can be used to collect from these systems. At the time of this writing, the following Linux distributions are considered fully supported by the SSH Collection Module:

  • RHEL/Oracle/CentOS 5.x, 6.x, 7.x
  • Ubuntu 12.04, 14.04, 16.04
  • SUSE Linux Enterprise Server 9, 10, 11, 12
  • IBM AIX

System Eligibility

For a system to be considered eligible for participation in the SSH Collection Module, a set of conditions must be met. Some of these conditions are common to all protocol types used in the engagement process, and some are specific to the SSH Collection Module:

Common Requirements

  • The system responds to an ICMP Echo Request (ping) sourced from the RN150 Virtual Appliance
  • The RN150 Virtual Appliance can communicate over TCP or UDP to the system
  • The protocol type has been opted-in to on the RN150 Virtual Appliance
  • At least one credential for the protocol has been entered in the RN150 Virtual Appliance

SSH Collection Module Requirements

  • The system has a running SSH server
  • The RN150 Virtual Appliance is permitted to communicate with the system via the server TCP port
  • The system firewall permits communication to the server TCP port from the RN150 Virtual Appliance IP
  • The SSH server is configured to allow connections from the RN150 Virtual Appliance
  • The SSH server is configured to allow the authentication type selected for the provided credential
  • The user account associated with the credential is a valid account on the system
  • The user account has a valid shell
  • The user account is permitted to utilize sudo, unless using the root account
  • sudo is configured according to the requirements listed below
  • The operating system is understood and supported by the SSH Collection Module


Credential Utilization

The SSH Collection Module utilizes a process common with other protocol types such as SNMP and WMI. When a system is discovered, meaning it responds to an ICMP ping and is found to have TCP port 22 available (or an alternate port, see section Custom Server Ports below), the SSH Collection Module will iteratively utilize each provided SSH credential to attempt to authenticate with the system. The first credential that is successful in connecting and authenticating, and for which other validation checks are successful, will be mapped to that device. Further communication with the system will be conducted using that particular credential entry.

This allows the user to enter a single credential that will be valid for use by any systems that are configured to utilize that credential, and the user will only need to enter each unique credential that is intended to be utilized. As the SSH Collection Module does not know at the outset which credential entry maps to which device(s), this may result in a number of failed authentication attempts as the SSH Collection Module tries each credential to derive the correct entry for a given system. An IPS or other system that monitors failed authentication attempts may be triggered by this behavior. In this case, an exception may need to be made for the purposes of the SSH Collection Module.


User Account Requirements

The SSH protocol authentication process centers around a user account. All forms of authentication will require a username. The username requirements of the SSH Collection Module module are the requirements of the SSH protocol, including but not limited to:

  • The user account must be a valid, known user account on the target system
  • The user account must be enabled
  • The user account must have a valid shell permitting login
  • The user account must be permitted to perform login via SSH

Some systems will not include the /sbin directory in the default command $PATH. For reference, the environment variable $PATH contains a colon-delimited list of directories that the shell searches for commands that are not provided using absolute paths. Some commands required by the SSH Collection Module are located in the /sbin directory, and if this is not included in the $PATH, attempts to utilize the commands will fail. It is highly recommended that the /sbin directory be included in the $PATH for the user account being utilized. Details on this are shown in the configuration examples below.


Authentication Types

The SSH protocol provides a number of different authentication mechanisms for providing alternative data, beyond the username, for authenticating with the SSH server. The following authentication mechanisms, described using the SSH terminology, are supported by the SSH Collection Module:

  • password
  • keyboard-interactive
  • publickey

The password and keyboard-interactive authentication types are both password based. The password type performs a direct user password validation facility, while the keyboard-interactive type uses the PAM infrastructure. Configuration on the RN150 Virtual Appliance for either type is identical, and the password string should be provided as a component of the credential entry.

The publickey authentication type uses asymmetric key pairs, most often RSA keys. In this model, the private key of the pair is utilized by the client, while the corresponding public key is attached to the user account on the server. As the SSH Collection Module on the RN150 Virtual Appliance is acting as the client, the private key of the pair is provided as a component of the credential entry, while the public key is a component of the user account configuration on the target systems. More details on enabling publickey authentication is provided below.


Key-based Authentication Requirements

There are a variety of differing key types supported by various SSH implementations. As the SSH Collection Module uses the OpenSSH distribution for the SSH client, the key type used must be understood by OpenSSH.

The requirements for keys provided as part of an SSH credential entry for the SSH Collection Module are as follows:

  • The private key must be in the ASCII PEM encoding, which is the default when generated using the ssh-keygen utility

 Certain key generators, will generate a binary-form key file. If such a key is desired to be used, it must first be converted to ASCII PEM format prior to use. Consult the documentation for the key generator software for details on key conversions. It is highly recommended that the conversion is performed on a copy of the key, and that the successful utilization of the key is validated prior to use as a credential entry for the SSH Collection Module.

Private keys generated by the PuTTY client popular on Windows platforms are not directly compatible with the OpenSSH client, although the OpenSSH server supports validation of PuTTY public keys. These keys will need to be converted to the OpenSSH format prior to use.


Privilege Elevation

Some commands utilized during the collection processes require elevated privilege on the target system. This corresponds to elevation to the root account. The specific commands requiring elevated privilege are provided below.

If the username associated with the credential entry is exactly root, then no additional privilege elevation will be attempted when issuing commands. Any other username involves the use of the sudo utility to perform the elevation. The sudo utility is ubiquitous in the Linux and UNIX-class system space, and will be provided by the operating system for many systems and/or distributions.

The use of sudo introduces several requirements for configuration, and is typically the most involved portion of configuring the environment for participation in the SSH Collection Module.

sudo is typically password-based, and was designed for interactive use. Particularly, an interactive session on a system will involve a terminal device, or TTY, that is associated with the login shell of the session. When using the SSH protocol as a communication transport in a non-interactive manner, a TTY device on the target system is not allocated. Due to the automated nature of the SSH Collection Module, this means that the SSH Collection Module will not allocate a TTY.

In order to provide sudo with a password, the utility typically requires a TTY device to present a password entry prompt. As the SSH Collection Module does not allocate a TTY, this facility is not currently supported. In order to configure sudo to participate in the SSH Collection Module, the requirement of an associated TTY device must be disabled, and sudo must be configured to not prompt for a password. Both of these configuration items can be set on a per-user basis, either on the existing user account associated with the credential entry, or as a component of a specific account created for the purposes of utilizing the SSH Collection Module as part of the RISC Networks engagement process. Details on how to configure sudo for these requirements are provided below.


Entering Credentials

In order to participate in the SSH Collection Module, SSH authentication credentials must be provided to the SSH Collection Module via the RN150 Virtual Appliance credential configuration dialog. As stated above, you will only need to enter each unique credential that is intended to be used, which will be utilized for each system for which that credential is found to be successful.

To enter an SSH credential in the RN150 Virtual Appliance, access the configuration application on the appliance by using the virtual console or by browsing to the appliance IP address via HTTP or HTTPS. From the Dashboard page of the application, select the SSH section. Please note that the SSH Collection Module and the Cisco CLI collection module are different technologies, and to utilize the latter you will need to access the Additional Credentials section, select CLI, and provide credentials there.

Accessing the SSH section will provide an input form for entering credential entries. The exact process of entering a credential varies slightly depending on the authentication type being used for that particular credential entry.

For all authentication types, first enter the username for the credential in the Username field.

Next, from the drop-down list entitled Auth Type, select the type of authentication desired. This will currently be one of password or publickey. The password type should be selected for either password or keyboard-interactive authentication.

If password is selected, an additional entry field will be presented, in which the password should be entered.

If publickey is selected, two input fields will be presented. The full text contents of the private key associated with the credential should be pasted in the Private Key text field. Please note that if the virtual console does not permit copy-and-paste, it may be necessary to access the appliance configuration application over HTTP or HTTPS in order to paste in the contents. The key contents must be PEM encoded ASCII text. The key contents will typically begin and end with a header field, which should be included in the entered text. If the private key being used in the credential entry is passphrase-encrypted, the passphrase associated with the key should be entered in the Key Passphrase field. If the key requires a passphrase and one is not provided, that key will not be able to be successfully used by the SSH Collection Module. If the private key is not passphrase-encrypted, then the Key Passphrase field should be left blank. If a passphrase is entered for an unencrypted key, this will not prevent the SSH Collection Module from successfully using the key, but may be confusing later on or to other users participating in credential entry. If a passphrase-encrypted key is entered without a passphrase and committed to the credentials list, editing that entry will allow the addition of a passphrase.

The Privilege Elevation field is currently not able to be explicitly set by the user, and the value of this field will be automatically determined based on the username provided. If the username is exactly 'root', then the value will be set to None, while any other username will cause the value to be set to sudo.

The Port field specifies the TCP port the client should connect to when utilizing the credential. Port 22 is the default, and will be automatically populated in the field. Any valid TCP port, with some exceptions, can be entered instead. See the Custom Server Ports section for more information.

Once the credential entry form is completed, select the Add button. This will present a further dialog, similar to other credential types. This dialog will present a text form that will be auto-populated with the IP address of the default gateway of the RN150 Virtual Appliance network interface. You can replace this IP address with the IP address of a known system that meets the eligibility requirements to test the operation of the credential to that system. It is highly recommended to test each credential that is entered, and if the credential is valid for a number of systems, it is further recommended to test to a sample of these systems. If the Test button is selected, the credential will be attempted against the provided IP address, and if successful, the credential will be added to the list of provided credentials. If the attempt is unsuccessful, the response from the target system will be shown in the dialog where you can either select a different IP address to test or select Cancel to alter the configuration of the credential. If the test is not desired, then Skip can be selected to immediately add the credential.

The SSH credential page will display some information about the credentials that have been entered. If an existing credential needs to be modified, then select Edit to open a configuration dialog where the values can be manipulated. Please note that manipulating an existing entry will change the credential for all systems in inventory that are mapped to the credential, and can result in a loss of communication with those system using the SSH Collection Module. If an existing credential needs to be tested to another known system, then you can select Edit, then without modifying the credential select Update. This will bring up the testing dialog described earlier.

If a credential needs to be deleted, then select the Delete button. Please note that deleting a credential will render it unusable for any systems that have been mapped to that credential, and can result in a loss of communication to those systems using the SSH Collection Module.

Custom Server Ports

The SSH Collection Module allows you to set an alternate TCP port for a given credential entry. The SSH protocol uses TCP port 22 as a standard, however some environments may run SSH servers on alternate ports. The TCP port selected when entering a credential is localized to that specific credential entry. Be aware, however, that when an alternate port is provided as a component of an SSH credential entry, that TCP port will be included in the discovery process, and that port will be checked for every IP address that responds during the ICMP portion of discovery to determine if that IP address supports SSH. If a device is found to have that TCP port open, then the device will be considered to support SSH.

Custom SSH ports provided for SSH credentials must meet a set of criteria to be accepted as a valid credential. To meet these criteria, a port must be numeric, within the range of valid TCP ports (1 - 65535), and not conflict with the standard ports utilized by other collection protocols. The following ports, and their related protocols, are not considered valid SSH ports: 80 (HTTP), 443 (HTTPS), 135 (WMI), 161 (SNMP), 62078 (iphone-sync).

Because user-supplied port numbers are automatically added to the list of ports used during the discovery process, a limitation on the number of unique custom ports is enforced. The credential configuration interface will produce an error indicating this limit if it is reached. The limit reflects the count of unique port numbers across all credential entries, not the number of credential entries using custom ports. When modifying the port on an existing credential, the new port number must also satisfy the custom port count limitation.

If a credential entry with a custom SSH port is removed, and the associated port number is not present on another credential entry that has not been removed, then the port will not be included in the discovery process and will not count towards the custom port count limitation.

Configuration Examples

DISCLAIMER: The configuration examples shown here are for informational and illustrative purposes only, and are not indented to be used directly. Always consult the documentation for your systems and fully understand the ramifications of configuration changes prior to implementation.

The following are some specific examples of how a system may be configured to participate fully in the SSH Collection Module. Your mileage with these examples will vary depending on the specifics of the system, the configuration requirements for the system or the environment, and other factors. All examples below assume that the user account being utilized is named 'risc', and the IP address of the RN150 Virtual Appliance is 10.0.0.2. The examples are provided in a pseudo-shell format, where lines prefixed with the '#' symbol represent comments about the operation being performed while those that do not begin with '#' are shell-like commands.

Creating A User Account On Linux

This is an example of how to create a user account on a Linux system. This is for illustrative purposes only. Consult the documentation for your system before attempting to create a user account.

1. Create the user account.

useradd -m risc

2. Set the user's password.

passwd risc

3. Configure the PATH and LC_ALL variables for the user account by modifying the shell initialization files.

echo 'export PATH=$PATH:/sbin' >> ~/.bashrc
echo 'export LC_ALL=C' >> ~/.bashrc


Creating A Key Pair

This is an example of how to create an SSH key pair. The example assumes that the user account used for collection is named 'risc'.

1. Log in to the account.

su - risc

2. Create a keypair. This example uses the default configuration. See the ssh-keygen manual for configuration options.

ssh-keygen

3. Add the public key to the list of keys allowing login to the local system.

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

4. Ensure the file permissions on the authorized_keys file are correct. SSH is particular about file permissions, and will refuse to work if they are incorrect.

chmod 600 ~/.ssh/authorized_keys

5. Retrieve the contents of the private key. This can be copied into the RN150 appliance when creating an SSH credential entry.

cat ~/.ssh/id_rsa


Testing Sudo Configuration

This is an example series of commands for testing the sudo configuration. It assumes that the user account used for collection is named 'risc'.

1. Log in to the account

su - risc

2. Run the built-in sudo configuration test to confirm that the sudoers syntax is valid and the user account has basic access to run sudo.

sudo -v

3. Run the true command under sudo to confirm that the account is permitted to execute a command. The true command is used by the SSH Collection Module to test access to sudo.

sudo true


Basic Example Sudo Configuration

This is an example configuration for a Linux system that broadly grants the ability to execute commands using sudo. This is the least secure option, but is easy to implement and maintain. It assumes that the user account used for collection is named 'risc'.

# remove the TTY restriction for this user
Defaults:risc !requiretty

# allow the user to issue commands as root without a password
risc ALL=(root) NOPASSWD: ALL

Advanced Example Sudo Configuration

This is an example configuration for a Linux system that explicitly defines the commands that are permitted to be executed through sudo. It assumes that the user account used for collection is named 'risc'. The commands are specified by their absolute paths, and may differ between OS distributions and/or versions. This example is for illustrative purposes only. It may not contain all commands requiring sudo. See the Command Reference below for an up-to-date list.

# remove the TTY restriction for this user
Defaults:risc !requiretty

# allow specific commands permitted for the user
# backslashes are used to continue the configuration on a new line for clarity
risc ALL=(root) NOPASSWD: \
			/bin/true, \
			/sbin/ifconfig, \
			/bin/df -P, \
			/bin/netstat --inet --inet6 -n -p -a -t, \
			/sbin/fdisk -l, \
			/bin/cat /sys/devices/virtual/dmi/id/*, \
			/usr/sbin/dmidecode --type system, \
			/usr/sbin/dmidecode --type chassis, \
			/usr/sbin/dmidecode --type bios


Command Reference

The following tables describe the commands that may be issued by the SSH Collection module. The commands that are issued depend on the operating system. Not all listed commands are issued against every system. Certain commands are only issued in the case that a more preferable command is not available, and certain commands may be specific to a particular OS distribution and/or version.

The Privileged column describes whether the command will be issued using the sudo utility to elevate privileges. Commands listed as 'fallback' are first attempted without sudo, and then attempted with sudo if the first attempt fails.

Linux

Command

Privileged

Command Operation

Reason for use

sudo true
YesImmediately returns successfully.

Used to validate access to the sudo utility.

uname -a

Returns all elements in the system utsname struct.Used for a full descriptive string for the system.
uname -s

Returns the operating system name.Used for OS detection.
uname -r

Returns the operating system release version.Used for kernel version .detection
uname -p

Returns the hardware platform type.Used for system architecture detection.
uname -m

Returns the hardware machine type.Used for system architecture detection.
uname -n

Returns the system hostname.Used for hostname detection.
cat /etc/os-release

Emits the contents of a file.Used during OS distribution detection.
cat /etc/oracle-release

Emits the contents of a file.

Used during OS distribution detection.

cat /etc/redhat-release

Emits the contents of a file.Used during OS distribution detection.
find /etc -type f | grep -e '[-_]release$' -e '[-_]version$'

Finds files matching a search pattern. Used during distribution detection, to find distribution information files not listed above.
cat $file

Emits the contents of a file.Used for to retrieve the contents of a distribution file discovered by the command above. The filename must be recognized before operating on it.
cat /proc/uptime

Emits the contents of a file.Used to retrieve the system uptime.
w -h
Returns details of logged in users.Used to retrieve the number of current user logins.
cat /proc/cpuinfo

Emits the contents of a file.Used to retrieve CPU topology.
cat /proc/meminfo

Emits the contents of a file.Used to retrieve system memory size and utilization.
ps axwww --no-headers -o pid,cputime,rsz,command

Returns details on running processes.Used to retrieve a list of running processes.
sudo ifconfig –a
YesReturns details on network interfaces.Used to retrieve details on all network interfaces.
sudo ifconfig $interface
YesReturns details on network interfaces.Used to retrieve details on a specific network interface, where $interface is the name of a previously collected network interface.
cat /sys/class/net/$interface/ifindex

Emits the contents of a file.Used to retrieve the network interface index number, where $interface is the name of a previously collected network interface.
cat /sys/class/net/$interface/operstate

Emits the contents of a file.Used to retrieve the network interface physical state, where $interface is the name of a previously collected network interface.
cat /sys/class/net/$interface/mtu

Emits the contents of a file.Used to retrieve the network interface MTU, where $interface is the name of a previously collected network interface.
cat /sys/class/net/$interface/speed

Emits the contents of a file.Used to retrieve the network interface bps rate, where $interface is the name of a previously collected network interface.
readlink /sys/class/net/$interface/device/driver/module

Canonicalizes the path of a symlink or file.Used to retrieve the network interface driver, where $interface is the name of a previously collected network interface.
lsblk -dnb --output NAME,MAJ:MIN,SIZE,MODEL

Returns details on disks and partitions.Used to retrieve physical disk details
sudo fdisk –l
YesLists or configures disk partitions.Used to retrieve physical disk details if lsblk is unavailable.
cat /sys/block/$device/device/model

Emits the contents of a file.Used to retrieve the disk model string, if available, where $device is the name of a previously collected disk device.
cat /proc/partitions

Emits the contents of a file.Used to retrieve disk partition details.
mountFallbackLists mounted filesystems.Used to retrieve a list of mounted filesystems.
sudo df -PYesLists filesystem utilization details.Used to retrieve filesystem utilization details.
vmstat –w –S K 1 2

Lists various system performance details.Used to retrieve system performance details. It may be used without the -w flag for some systems. The command produces two metric reports with a one second wait between them.
cat /proc/diskstats

Emits the contents of a file.Used to retrieve disk performance details.
sudo netstat --inet --inet6 -n -p –a -t
YesLists open sockets.Used to retrieve network connections.
cat /proc/net/dev

Emits the contents of a file.Used to retrieve various network subsystem statistics.
ls /sys/devices/virtual/dmi/id

Lists directory contents.Used to check for the existence of hardware platform data using sysfs.
cat /sys/devices/virtual/dmi/id/sys_vendor

Emits the contents of a file.Used to retrieve the system hardware vendor from sysfs.
cat /sys/devices/virtual/dmi/id/product_name

Emits the contents of a file.Used to retrieve the system product name from sysfs.
cat /sys/devices/virtual/dmi/id/product_version

Emits the contents of a file.Used to retrieve the system product version from sysfs.
sudo cat /sys/devices/virtual/dmi/id/product_serial
YesEmits the contents of a file.Used to retrieve the system product serial number from sysfs.
sudo cat /sys/devices/virtual/dmi/id/product_uuid
YesEmits the contents of a file.Used to retrieve the system product UUID from sysfs.
cat /sys/devices/virtual/dmi/id/chassis_vendor

Emits the contents of a file.Used to retrieve the chassis vendor from sysfs.
cat /sys/devices/virtual/dmi/id/chassis_version

Emits the contents of a file.Used to retrieve the chassis version from sysfs.
sudo cat /sys/devices/virtual/dmi/id/chassis_serial
YesEmits the contents of a file.Used to retrieve the chassis serial number from sysfs.
cat /sys/devices/virtual/dmi/id/bios_vendor

Emits the contents of a file.Used to retrieve the BIOS vendor from sysfs.
cat /sys/devices/virtual/dmi/id/bios_version

Emits the contents of a file.Used to retrieve the BIOS version from sysfs.
cat /sys/devices/virtual/dmi/id/bios_date

Emits the contents of a file.Used to retrieve the BIOS build date from sysfs.
which dmidecode

Lists the location of an executable file.Used to check for the existence of the dmidecode utility, as a fallback if sysfs is unavailable.
sudo dmidecode --type system
YesRetrieves hardware info from DMI.Used to retrieve system hardware product information.
sudo dmidecode --type chassis
YesRetrieves hardware info from DMI.Used to retrieve chassis hardware information.
sudo dmidecode --type bios
YesRetrieves hardware info from DMI.Used to retrieve BIOS information.
(rpm -qf /bin/sh >>/dev/null 2>&1 && echo rpm) || (dpkg -S /bin/sh >>/dev/null 2>&1 && echo dpkg) || echo none
Queries for installed package metadata.Used to detect a system's primary package manager by querying for the owner of the /bin/sh file.
rpm -qa --queryformat "META^|^%{NAME}^|^%{EPOCH}^|^%{VERSION}^|^%{RELEASE}^|^%{SUMMARY}\n[%{FILENAMES}\n]/ / /\n"|grep -E '^META\^\|\^|^/ / /$|^/.*bin/'
Lists metadata for installed software packages.Used to retrieve a complete list of installed packages and their executables for systems using rpm as a package manager.
dpkg-query --show --showformat='META^|^${binary:Package}^|^${Version}^|^${db:Status-Abbrev}^|^${binary:Summary}\n'
Lists metadata for installed software packages.Used to retrieve a complete list of installed packages for systems using dpkg as a package manager.
dpkg-query -S 'bin/'
Lists metadata for installed software packages.Used to retrieve a list of executables installed on the system and the package that installed that executable for systems using dpkg as a package manager.


AIX

Command

Privileged

Command Operation

Reason for use

sudo true
YesImmediately returns successfully.Used to validate access to the sudo utility.
uname -a

Returns all elements in the system utsname structure.Used for a full descriptive string for the system.
uname -s

Returns the operating system name.Used for OS detection.
uname -r

returns the operating system minor versionUsed for OS version detection.
uname -v

returns the operating system major versionUsed for OS version detection.
uname -p

Returns the hardware platform type.Used for system architecture detection.
uname -n

Returns the system hostname.Used for hostname detection.
lsdev

Returns a list of devices on the system.Used to determine devices eligible for further inspection.
lsdev -Cc if

Returns a list of network interfaces on the system.Used to retrieve list of network devices.
ps -o etime= -p1

Returns details on running processes.Used to derive the system uptime, by determining the elapsed time since the init process started.
w -h

Returns details of logged in users.Used to retrieve the number of current user logins.
lsattr -E -l sys0 -a realmem

Returns details on a system logical device.Used to determine the size of system memory.
ps -e -o pid,cputime,rssize,args

Returns details on running processes.Used to retrieve a list of running processes.
lsattr -E -l procN

Returns details on a system logical device.Used to retrieve CPU details, where N is a CPU device index seen from lsdev.
ifconfig -a

Returns details on network interfaces.Used to retrieve details on network interfaces.
entstat $device

Returns statistics on an ethernet device.Used to retrieve statistics for ethernet devices, where $device is the name of an ethernet device.
lsattr -E -l $device -a state,netaddr,netmask,netaddr6,mtu

Returns details on a system logical device.Used to retrieve statistics for ethernet devices, where $device is the name of an ethernet device.
sudo getconf DISK_SIZE /dev/$device
FallbackReturns the value of a system configuration variable.Used to retrieve the size of physical disks reported by lsdevThe disks may have access restrictions in some configurations that require privilege elevation.
lsfs
FallbackReturns filesystem details.Used to retrieve a list of filesystems.
sudo df -Pk
FallbackReturns filesystem details.Used to retrieve filesystem size and utilization. Normally, df doesn't require elevated privileges. However, in some cases, certain file-systems cannot be shown without, and if df returns an error, it is attempted with elevated privileges.
lslpp -lJq | grep -v '^#Path'

Returns a list of installed software.Used to retrieve a list of installed software, using grep to remove header lines.
vmstat 1 2

Returns details on system utilization.Used to retrieve various system utilization statistics. It produces two metric reports, with a one second wait between them.
vmstat -v

Returns details on virtual memory utilization.Used to retrieve detailed information on virtual memory utilization.
pagesize

Returns the size of a virtual memory page.Used to convert virtual memory page utilization to byte-sizes.
iostat -Dl 1 1

Returns statistics on system I/O activity.Used to retrieve statistics on system I/O.
sudo lsof -i -nP
FallbackReturns a list of open file descriptors.Used to retrieve details of open network sockets. Normally, lsfs doesn't require privilege elevation, however, in some cases it seems it may. To avoid interfering with engagements where this is working and not configured for privilege elevation we first try without and then fall back to try with.

Troubleshooting

Troubleshooting Introduction

It is highly recommended to test SSH access and operation prior to issuing a Discovery scan, to ensure that the hosts are configured correctly for the SSH Collection Module. This can be performed from the SSH credentials section on the RN150 Virtual Appliance, which will validate the ability of the RN150 Virtual Appliance to connect to the target hosts at the network level, and will test the native operation of the SSH Collection Module. However, it may be desired to perform more manual tests from another host in the environment, allowing the enablement of debugging options in SSH for deeper analysis.

Testing from the RN150 Virtual Applaince is covered above in the credential entry sections. This section will cover the manual tests that can be performed for deeper analysis of the configuration.

To test SSH access to a target host, first log in to another host in the environment that provides an SSH client. This will be referred to as the "local" host, while the host that is being connected to is referred to as the "target". In the following examples, the target is assumed to have an IP address of 10.0.0.2, and the username of the account we are connecting to is "risc".

The general structure of the SSH command, in manpage format, is as follows:

    ssh [flags] [authentication-flags] <username>@<host-IP> "<command>"

A breakdown of this command:

Component

Format

Description

ssh
literalThe ssh client binary
[flags]
optionalGeneral flags to control the ssh client behavior
[authentication-flags]
optionalControls explicit authentication behavior
<username>
replace

The username of the account we are connecting with

<host-IP>
replaceThe IP address of the target host
"<command>"
replaceThe command to issue on the target host, contained in single or double quotes

Troubleshooting Command Reference

Normally, the ssh client is invoked without the final argument shown above, which performs a login to the target host into an interactive shell session. When invoked with the optional command argument, this causes the ssh client to only invoke that command in a non-interactive fashion. This is an important component of the troubleshooting process, as the use of the command argument will replicate the behavior of the SSH Collection Module. It also provides a test for the behavior described above; an interactive session will allocate a TTY terminal device, while the non-interactive execution of a single command will not. This allows testing the sudo configuration, which will often require a TTY device associated with the user session invoking it.

When performing troubleshooting to determine the cause of an authentication failure experienced by the SSH Collection Module, the '-v' flag can be given to the ssh client to cause it to emit extra verbose debugging output. This flag can be given multiple times to cause the ssh client to emit more information. It is highly recommended to use this flag when performing troubleshooting.

Depending on the type of authentication being utilized, the authentication-flags field may need to be used to cause the client to specifically use the type of authentication desired, as well as the specific authentication credentials to be used.

If using password or keyboard-interactive authentication, this can be specified as, for example:

    -o PreferredAuthentications="password"

If using publickey authentication, this would be:

    -o PreferredAuthentications="publickey"

If using publickey authentication, to specify which key the client utilizes for authentication attempts, the following could be used, for example:

    -i /home/risc/key


To perform a simple connection test using password authentication to the target host to validate the connection, authentication, and ability to issue a simple command:

    ssh -v -o PreferredAuthentications="password" risc@10.0.0.2 "true"


To test the same with publickey authentication:

    ssh -v -o PreferredAuthentications="publickey" -i /home/risc/key risc@10.0.0.2 "true"


 Testing the availability and configuration of sudo is of primary importance, as this is often the main reason for a failure to bring a host into inventory.

    ssh -v risc@10.0.0.2 'sudo ifconfig -a'

Troubleshooting Command Suite

The following is a series of commands that will validate the enablement of the SSH Collection Module for a given host. When a support case is opened, a support engineer will likely request the results of the follow series of commands. In this case, it is not expected that legitimate customer data that results from a successful command be sent to support, but any error messages produced will be expected to be provided. These error messages may be cleaned to the extent that usernames or other data deemed private may be removed, however extensive redaction of data may result in difficulty providing insight into the failure.

In these examples, each ssh command is followed by a shell command to print the exit status of the previously issued command. Replace the user and IP elements in angle brackets with the username being utilized and the IP address of the target host.

For Linux targets:

ssh -v <user>@<IP> 'true'
echo $?
ssh -v <user>@<IP> 'sudo true'
echo $?
ssh -v <user>@<IP> 'uname -s'
echo $?
ssh -v <user>@<IP> 'sudo ifconfig -a'
echo $?
ssh -v <user>@<IP> 'sudo df -P'
echo $?
ssh -v <user>@<IP> 'sudo netstat --inet --inet6 -n -p –a -t'
echo $?

For AIX targets:

ssh -v <user>@<IP> 'true'
echo $?
ssh -v <user>@<IP> 'sudo true'
echo $?
ssh -v <user>@<IP> 'uname -s'
echo $?
ssh -v <user>@<IP> 'lsdev'
echo $?
ssh -v <user>@<IP> 'lsattr -E -l sys0 -a realmem'
echo $?
ssh -v <user>@<IP> 'df -Pk'
echo $?
ssh -v <user>@<IP> 'df -Pk'
echo $?
ssh -v <user>@<IP> 'lsof -i -nP'
echo $?
ssh -v <user>@<IP> 'netstat -an -f inet'
echo $?

Error Messages

Understanding the error messages received when testing the credential, through the RN150 Virtual Appliance credential testing facility or through manual testing, is key to pinpointing the problem. Due to the nature of the ssh client/server interaction, it is common to receive non-error related text in the error message. This includes any banner text returned to the client by the server, as well as a message indicating that the server's host keys have been added to the list of known hosts. These elements of the error can be ignored.

The following is a non-exhaustive list of common error messages, and their meaning:

Error Text

Context

Description of behavior

Actions

Failed to determine operating system using command 'uname -s'credential testing

The 'uname -s' command was not successful. The target device does not support this command, or the command was otherwise unable to execute. The successful completion of this command is a strict requirement for participation in the SSH Collection Module.

  • Confirm the target operating system is supported by the SSH Collection Module
  • Ensure the 'uname' command exists and the user account can execute it
  • Configure the device for SNMP collection
Unsupported Operating Systemcredential testingThe target operating system is not supported; the target device is not eligible for participation in the SSH Collection Module
  • Review the supported operating systems above
  • Configure the device for SNMP collection
Successfully connected, but sudo validation failedcredential testingConnection and authentication was successful, but the user account was unable to utilize sudo
  • Review sudo configuration
  • Review the sudo logs
Connection refusedconnection

The ssh client connection was not successful. This could mean that:

  • the ssh server is not running
  • the ssh server is not listening on TCP 22, or the custom TCP port configured for the credential
  • the local firewall on the host blocked the attempt
  • network ACLs blocked the attempt
  • Ensure ssh server is running on host
  • Ensure ssh server is listening on expected TCP port on the host
  • Ensure local firewall is not blocking
  • Ensure network ACLs are not blocking
No route to hostconnectionThe RN150 has no route to the target IP address
  • Review network configuration on the RN150
  • Review VLAN configurations
  • Review routing to that subnet
Permission denied (...)authentication

The credentials offered by the client were refused by the server. This could mean that:

  • the username is not a valid user on the target
  • the user is not permitted
  • the password is invalid for the user
  • the key was corrupted or an invalid or unknown type
  • the key was not valid for the user.
  • file permissions on files related to authentication (keys, configuration files, etc) were not correct

This message is followed by a list, in parentheses, of authentication methods announced by the server

  • Ensure credential was entered correctly in the RN150
  • Ensure username exists on the target
  • Review ssh access controls
  • Review authentication configuration
  • Ensure private key is not encrypted
  • Review file permissions on target
Connection reset by peerany

The target ssh server suddenly aborted the connection. This is a generic error that has a large number of possible causes.

Often, this is a particular form of refused connection, where the initial stages of the connection were successful but additional access controls determined that the connection cannot continue.

This could also be due to one of the following:

  • The TCP session was lost
  • The client and server were unable to negotiate compatible session keys and MACs
  • The server only supports version 1 of the SSH protocol
  • Server is unable to handle an additional connection, or to fork a process for the requested command
  • Ensure that hosts.deny, hosts.allow, or similar are configured to permit the connection
  • Ensure that the ssh server supports version 2 of the protocol
  • Review server logs on the target
sudo: sorry, you must have a tty to run sudoprivileged command executionThe sudo configuration requires a TTY device for the user session
  • Review sudo configuration documentation above
  • Apply configuration parameters to sudo
sudo: no tty present and no askpass program specifiedprivileged command executionThe sudo configuration does not require a TTY device, but requires a password. Due to the necessity of having a TTY device in order to provide a password prompt, this results in sudo refusing to operate
  • Review the sudo configuration
  • Remove the password restriction for the user account

sudo: >>> /etc/sudoers: syntax error near line #<<<
sudo: parse error in /etc/sudoers near line #
sudo: no valid sudoers sources found, quitting
sudo: unable to initialize policy plugin

privileged command executionThe sudo configuration is syntactically invalid and was refused
  • Correct sudo configuration syntax
  • Test sudo locally on target