iLO BLADE SERVER SETUP AUTOMATION SCRIPTS

Frank da Cruz
Columbia University
As of: ilosetup 1.15 / Mon Feb 7 13:14:21 2011

Photo by Megan Pengelly-Anderson

Columbia University, like many other large institutions and companies, is replacing older computers with thin, rackmountable, interchangeable, modular computers called blades, in our case the ones from Hewlett-Packard, ProLiant 64-bit processors capable of running a variety of operating systems, including Windows and Linux.

The HP blade servers are set up and configured through an "Onboard Administrator" called Integrated Lights-Out (iLO), which is part of the machine itself, and not dependent on any operating system or other software. A Web-based GUI management tool is provided for manual configuration, testing, and monitoring of the blades, and there is also a text-mode "back door" accessible by Telnet (in iLO 2 but not iLO 3 and later) or SSH that provides a traditional command-line management interface.

When a company finds itself installing dozens or hundreds of blade servers, manual configuration through the Web interface is prohibitively repetitive, labor-intensive, and error-prone.

The task can be handled more efficiently and with less risk by using a program that automates the configuration dialog with each blade to yield a consistent configuration across any number of blades. A good tool for doing this is C-Kermit because it is designed specifically for automating dialogs across communications connections, including Telnet and SSH.

Two iLO configuration scripts have been written for C-Kermit, to be used on any reasonably modern Unix-based computer (Linux, Mac OS X, NetBSD, Solaris, HP-UX, AIX, etc). These scripts have been used to set up the 200 blade systems that have arrived so far and, according to the Senior Systems Engineer in charge of the deploying the blades, "we plan on using it for all future HP purchases - blade and otherwise. The primary advantages of this script are that we no longer need to attach a monitor and keyboard to each machine to manually configure the iLO – adding users, enabling ssh, configuring hostnames."

The main script, ilosetup, configures a blade (or any other HP server that has iLO). It needs to know the IP hostname for the blade and the username and password to create on the blade. It can also be given various option parameters.

The second script is ilodomany which simply executes ilosetup a number of times in sequence; once for each hostname given on the command line. This is so the created user ID and password do not need to be hardwired into the script or stored on a disk, or entered manually for each server. Instead, the access info is given once to ilodomany and then passed to each instance of ilosetup in a relatively secure manner.

The Serials File

Photo by Stew Feuerstein

Each line in the serials file contains the following fields, separated by whitespace:

1. IP Hostname
2. Serial number
3. iLO MAC address (hex)
4. eth0 MAC address (hex)
5. eth1 MAC address (hex)
6. factory-set iLO access password
7. Warranty

By convention, the iLO Console must be registered as hostname-console. So, for example, if the server's IP name is SERVER123.CC.COLUMBIA.EDU then the iLO Console's IP name is SERVER123-CONSOLE.CC.COLUMBIA.EDU.

How to Use the iLO Setup Script

If you start ilosetup without any command-line arguments you get a usage message like this one:

Usage: [ ilo_username=x ] [ ilo_userpass=y ] ilosetup hostname [ options ]
Options:
  1. bios
  2. connect
  3. debug
  4. help
  5. iloonly
  6. nopassword
  7. nopoweroff
  8. noraid
  9. noreset
 10. notelnet
 11. poweroff
 12. pwrmode
 13. quiet
 14. raid
 15. reset
 16. ssh
 17. telnet
 18. usepassword
 19. verbose
Default options: verbose nopoweroff telnet reset usepassword raid
Optional environment variables:
 ilo_userid:   user ID to create
 ilo_userpass: password for user ID
 DEBUG=1: turn on debugging
If created user ID and password are not found in the environment,
you are prompted for them.

Example 1:

ilosetup server123 nopoweroff reset

Example 2:

ilosetup server456 pwrmode:3

Example 3:

ilosetup server789 ssh connect    (makes an interactive SSH terminal connection to the iLO prompt)

To pass options from ilodomany to ilosetup, include an options clause on the ilodomany command line:

options="someoption anotheroption someotheroption"

Example using ilodomany to configure several hosts the same way:

ilodomany server123 server124 server125 options="nopoweroff reset"

The password options, usepassword and nopassword, are because sometimes a blade is delivered with a blank console password instead of the password that the manufacturer claims to have assigned. The 'usepassword' option (default) means to try the factory-set password taken from the serials file first, and if that doesn't work, to try a blank password; the 'nopassword' option means try a blank password first, and if that doesn't work, try the password from the serials file.

As noted, the created username and password are picked up from environment variables ilo_username and ilo_password, if defined; otherwise the script prompts for them. Then it looks up the host in the serials file to get the iLO access password. Then it makes an SSH connection to the blade and runs through the configuration dialog, checking each operation for success or failure, and logging everything that happens. At Columbia, the configuration goes as follows:

1. A session log is opened to record the dialog.
2. A Telnet (or SSH) connection is attempted.
3. If this fails, the other kind of connection is attempted.

   As delivered, the blades have Telnet disabled and SSH enabled. However some problems have been noted with the iLO SSH implementation so Telnet is preferred here. Therefore the first time the script is run on a blade, SSH is used, but then the script itself enables Telnet, so subsequent accesses can be with Telnet.

   iLO 3 does not support telnet. However, unless instructed otherwise ilosetup tries first to connect with telnet and when this fails, it tries SSH. The script still works but an unnecessary and fruitless step is taken on iLO 3 boxes. Include the 'notelnet' option on the command line to avoid it. If desired the script can be changed to always try SSH first.

4. If no connection was made the script writes a message to the log and exits.
5. The script attempts to log in to the iLO console; upon failure exits with failure status code.
6. Once logged in, the script prompts the user for the created ID and password if these weren't found in the environment.
7. With the created user ID and password in hand, the script issues iLO commands to:
   • Capture the iLO firmware version
   • Create the specified user ID
   • Enable Telnet access (iLO 2 and below)
   • Enable a shared console
   • Set the internal server name to match the IP hostname
8. If the iloonly option was given, the script exits (first resetting iLO, unless the noreset option was given). Otherwise it proceeds with the following steps, which power the server off and on (which will, of course, disrupt any work that is being done on it).
9. Now we check to see if the host is powered on or off. If it's on, we shut it down and power it up again (otherwise we just power it up) in order to be able to access the BIOS. Then we wait up to 100 seconds for an indication from the server that it actually is on, and if this doesn't come we fail and exit.
10. Once the server is on we enter the Virtual Serial Port (VSP) subsystem. This can fail for a variety of reasons, all of which are checked for and handled appropriately.
11. If the raid option was given, we enter the Arrays CLI, and if the blade has two physical disks, and it is not already configured to have a single logical RAID 1 drive, we configure it to have one, and then we return to VSP.
12. From VSP enter the RBSU subsystem, which is the BIOS, or fail and exit if we can't.
13. In the BIOS we issue a series of commands at the RBSU prompt to configure a virtual serial port, a serial console, and an EMS console, checking for errors along the way. The connection from the iLO console to VSP to RBSU is not well flow controlled, so the rate at which we send characters has to be throttled.
14. We exit from RBSU, "escape back" to the iLO console by sending the iLO's escape sequence.
15. If "poweroff at end" was requested, we power off the server.
16. Finally we reset iLO itself, so the iLO changes will take effect, exit, and close the connection.

The exit status is 0 if the script succeeded, nonzero if it failed.

Logs

xxx-connect.log
xxx-session.log

The logs are regular Kermit session logs, transcripts of characters that were received from the server during the session, intermixed with lines that start with “%%” that are progress messages from the script itself, which can be grepped for a quick summary of the run:

grep ^%% fillmore-session.log

In C-Kermit 8.0, the logs contain everything that came in, including escape sequences. With C-Kermit 9.0, the escape sequences are stripped out.

Implementation Notes

ilosetup	ilodomany

This script was written for C-Kermit 8.0.212 Dev.28, an internal working copy that is to be released as C-Kermit 9.0, which is much better adapted to modern platforms than the previous release (8.0.211 of April 2004). The iLO scripts use several features that are new to this version:

FSEEK /FIND:string	Seeks to the spot in a file opened by FOPEN that contains the specified string. In earlier Kermit versions this would be accomplished by reading the file a line at a time and checking each line for the string.
The \fkwval() function	This is a handy way to parse strings of the form foo=bar; it creates a macro named foo with the value bar (foo could be any single word, bar can be anything at all, and "=" is the default separator but others, such as ":" can be specified).
The PUTENV command	This is what allows ilodomany to pass the created user ID and password to ilsosetup in a secure manner: they are not stored in any file, nor do they appear on the command line where they might be seen with the Unix 'ps' or 'w' commands.

The \fkwval() function is used to parse status messages resulting from iLO commands:

status=2
status_tag=COMMAND PROCESSING FAILED
error_tag=COMMAND SYNTAX ERROR

status=0
status_tag=COMMAND COMPLETED
Global settings modified.

These new features are not yet in Kermit 95 (the Windows version of C-Kermit) because they were added after the most recent K95 release. In any case, as written, the scripts might depend on some Unixisms.

iLO and RBSU commands are issued by macros that take care the error handling according to each command, because not all errors should be fatal; for example if the created ID already exists, that's not a fatal error. The RBSU command macro, as noted, takes extreme measures to get the command to its destination despite the traps and pitfalls of the multi-leg communication pathway.

As always, C-Kermit depends on the external ssh client to make SSH connections. All modern Unixes should have this client; if not, it can be downloaded from here. C-Kermit's Telnet client is built-in.

iLO 3

The ilosetup script was adapted to these changes in such a way that it still works with iLO 2. When the connection is made, the script extracts the iLO version number from the greeting herald, and uses it to decide when do something special for iLO 3.

Adapting the scripts to Different Sites

$ chmod +x ilosetup ilodomany

#!/usr/local/bin/kermit +

To use the iLO scripts at a new site:

• Create a Serials File listing the blades.
• Modify any site-specific definitions in the script, which are gathered near the top, e.g. defaults, pathnames, domains, etc.
• If you want the script to perform different actions than those described, you'll need to modify the script; see the C-Kermit documentation.