IBM x3550 - Debian Squeeze Installation via IPMI Serial Over Lan

How to make it happen

Author: Nate Coraor <nate@psu.???>
Version: 0.2
Copyright: Creative Commons Attribution Non-Commercial Share Alike

0. Preface

This document is an attempt at step-by-step instructions for installing Debian Squeeze on an IBM x3550 using IPMI's Serial Over LAN console abilities. It also assumes you're going to install Debian via a PXE netboot. These instructions are valid for Debian's other boot methods, but since you're cool enough to use IPMI, you should be using netboot too.

I have not found a way to get into the BIOS over IPMI SOL. It seems you should be able to use ipmitool thusly:

% ipmitool -I lanplus -H bmc.example.org chassis bootdev bios

However, the BMC rejects this command:

Set Chassis Boot Parameter 5 failed: Invalid data field in request

And since neither ipmitool nor ipmiconsole seem to have a way to send the appropriate escape sequences to hit <F1>, it seems you're SOL for getting the BIOS over SOL (lawl).

You can force a PXE boot, though, which is explained later.

1. Get the BMC on the network

To configure the Baseboard Management Controller (BMC), you're probably going to need a monitor and keyboard. This is because the BMC is disabled from the factory, and I say probably because (if you're really motivated) you could create a USB drive that boots Linux and enables it from within the OS. But we're going to assume you're not going to do that.

The x3550's net1 is a shared BMC/host port. Note that the BMC is available in all x3550s. There is a seperate (optional) Remote Supervisor Adapter that uses the dedicated SM port. This document does not cover usage of the RSA and SM port.

2. BIOS/BMC Configuration

  1. Boot up and enter BIOS setup:

When the hardware boots, you will be presented with the opportunity to enter BIOS Setup:

IBM BIOS - (c) Copyright IBM Corporation 2007                             CP: 34
Symmetric Multiprocessing System
Intel(R) Xeon(R) CPU X5460 @ 3.16GHz
2 Processor Packages

32768 MB Installed Memory

Press F1 for Setup
Press F2 for Diagnostics
Press F12 to select boot device

>> BIOS Version 1.07 <<

You'll want to strike <F1> with gusto until the screen clears and Booting to Setup. appears.

  1. Enable Remote Console Redirection:

When the BIOS configuration loads, select Devices and I/O Ports. Set Serial Port A to Auto configure:

_

              ********************************************
              *          Devices and I/O Ports           *
              ********************************************
              *   Serial Port A     [ Auto*configure  ]  *
              *   Serial Port B     [ Disabled        ]  *
              *   Remote Console Redirection             *
              *                                          *
              *   Planar Ethernet 1 [ Enabled  ]         *
              *   Planar Ethernet 2 [ Enabled  ]         *
              *   Planar SAS        [ Enabled  ]         *
              *   PCI Slot 1        [ Enabled  ]         *
              *   PCI Slot 2        [ Enabled  ]         *
              *                                          *
              *                                          *
              *   Video                                  *
              *   IDE Configuration Menu                 *
              *   System MAC Addresses                   *
              ********************************************

Next, select Remote Console Redirection and set as such:

_
              ********************************************
              *          Devices and I/O Ports           *
        ********************************************************
        *              Remote Console Redirection              *
        ********************************************************
        *   Remote Console Active             [ Enabled  ]     *
        *   Remote Console COM Port           [ COM 1 ]        *
        *   Remote Console Baud Rate          [ 57600  ]       *
        *   Remote Console Data Bits          [ 8 ]            *
        *   Remote Console Parity             [ None ]         *
        *   Remote Console Stop Bits          [ 1 ]            *
        *   Remote Console Text Emulation     [ VT100/VT220 ]  *
        *   Remote Console Keyboard Emulation [ VT100/VT220 ]  *
        *   Remote Console Active After Boot  [ Enabled  ]     *
        *   Remote Console Flow Control       [ Disabled ]     *
        ********************************************************
              *   System MAC Addresses                   *
              ********************************************

You may want to leave the Text Emulation and Keyboard Emulation options as ANSI, which, if you're lucky, will draw nice ANSI pipe boxes instead of these ugly asterisks. Oddly enough this works fine for me on Dell PE1950s, but never draws correctly on the x3550. YMMV.

Back out to the main BIOS screen by striking <ESC> with hasty abandon.

  1. Assuming you're going to PXE boot, enable PXE on Planar Ethernet 1:

Select Start Options.

The x3550 doesn't support 802.1q VLAN Tagging on the management interface, so you're going to have to either use the 2nd interface for the OS, or put both the management and OS on the same subnet. Choose Planar Ethernet 1 or 2 accordingly, here:

_

      ************************************************************
      *                      Start Options                       *
      ************************************************************
      *   Startup Sequence Options                               *
      *                                                          *
      *   Planar Ethernet PXE/DHCP  [ Planar Ethernet 1       ]  *
      *   PCI Device Boot Priority  [ Planar SAS ]               *
      *   Keyboard NumLock State    [ Off ]                      *
      *   USB Disk                  [ Enabled  ]                 *
      *   Boot on POST/BIOS Error   [ Enabled  ]                 *
      *   Boot Fail Count           [ Enabled  ]                 *
      *   Rehook INT 19h            [ Disabled ]                 *
      *   Virus Detection           [ Disabled ]                 *
      ************************************************************

Back out to the main BIOS screen with an <ESC>.

  1. Configure the BMC

Select Advanced Setup, then select Baseboard Management Controller (BMC) Settings. It should look like this:

_

      **********************************************************
      *     Baseboard Management Controller (BMC) Settings     *
      **********************************************************
      *   BMC Firmware Version            01.34                *
      *   BMC Build Level                 GFBT50A              *
      *   BMC Build Date                  11*21*07             *
      *                                                        *
      *   BMC POST Watchdog               [ Disabled ]         *
      *   BMC POST Watchdog Timeout       [ 5 Minutes  ]       *
      *   System*BMC Serial Port Sharing  [ Enabled  ]         *
      *   BMC Serial Port Access Mode     [ Shared    ]        *
      *   Reboot System on NMI            [ Enabled  ]         *
      *   Power Restore Policy            [ Always On      ]   *
      *   BMC Fan Acoustic Mode           [ Standard ]         *
      *   BMC Network Configuration                            *
      *   BMC System Event Log                                 *
      *   User Account Settings                                *
      **********************************************************

Select BMC Network Configuration and configure it for your network. Examples, of course, your configuration will be different.

_

      **********************************************************
      *     Ba********************************************     *
      *********        BMC Network Configuration         *******
      *   BMC ********************************************     *
      *   BMC *   BMC MAC Address 08*00*20*C0*FF*EE      *     *
      *   BMC *   Host Name       [ 8020C0FFE ]          *     *
      *       *   DHCP Control    [ Use Static IP ]      *     *
      *   BMC *                                          *     *
      *   BMC *   IP Settings                            *     *
      *   Syst*   IP Address      [ 010.001.001.016  ]   *     *
      *   BMC *   Subnet Mask     [ 255.255.255.000  ]   *     *
      *   Rebo*   Gateway         [ 010.001.001.001  ]   *     *
      *   Powe*                                          * ]   *
      *   BMC *                                          *     *
      *   BMC *   Save Network Settings in BMC           *     *
      *   BMC ********************************************     *
      *   User Account Settings                                *
      **********************************************************

Then select Save Network Settings in BMC.

  1. Create an IPMI user:

Select User Acccount Settings:

_

      **********************************************************
      *     Baseboard Management Controller (BMC) Settings     *
      **********************************************************
      *   BMC Firmware Version            01.34                *
      *   **************************************************** *
      *   *              User Account Settings               * *
      *   **************************************************** *
      *   *   UserID#  Username         Enabled/Disabled     * *
      *   *   UserID 1 NULL                 Disabled         * *
      *   *   UserID 2 root                 Enabled          * *
      *   *   UserID 3 UNUSED               Disabled         * *
      *   *   UserID 4 UNUSED               Disabled         * *
      *   **************************************************** *
      *   BMC Fan Acoustic Mode           [ Standard ]         *
      *   BMC Network Configuration                            *
      *   BMC System Event Log                                 *
      *   User Account Settings                                *
      **********************************************************

Disable NULL and create a new user with Administrator privileges. You can figure that out without me pasting screens, right?

Make sure you select <<Save User Account Settings to BMC>> rather than just backing out with <ESC>.

Once users are configured, <ESC> out ot the main BIOS menu.

  1. Save and exit:

Select Save Settings to pop up the save dialog, and dismiss it.

Strike <ESC> to pop up the Exit menu, and select Yes ....

The host reboots.

4. Set up DHCP and netboot

netboot is the image used to PXE boot and install Debian. Note that net'inst' is not net'boot'!

  1. Set up a TFTP server somewhere.
  2. Set up a DHCP server somewhere. Doesn't have to be the same physical server as TFTP.
  3. Download netboot.tar.gz from the testing Debian Installer page and unpack it in (probably a subdirectory of) the TFTP server's directory (often /tftpboot or on Debian, /var/lib/tftpboot).
  4. Let's assume I stuck netboot in /tftpboot/debian/squeeze/amd64. Modify the pxelinux config in pxelinux.cfg/default, I generally just wipe out the contents and put what I'm actually going to install in there:
default menu.c32
prompt 0

menu title PXE Boot Menu

label squeeze-eth0-ttys0
    menu label ^Squeeze via eth0/ttyS0
    kernel debian/squeeze/amd64/linux
    append locale=en_US.UTF-8 interface=eth0 console-tools/archs=skip-config console-keymaps-at/keymap=us hostname=x domain=x preseed/url=http://foo.bar.baz/preseed-squeeze.cfg console=ttyS0,57600n8 DEBIAN_FRONTEND=text initrd=debian/squeeze/amd64/initrd.gz -- quiet

DEBIAN_FRONTEND is optional. You may have success with the default curses interface, but if you have all kinds of screen draw problems, switch to text. The IPMI SOL console seriously blows, and especially the implementation on the x3550 (maybe I should update the BMC firmware?).

  1. Add bnx2 firmware to the initrd image as explained here. Otherwise, your installer won't be able to use the network devices. And that's kinda useless.
  2. Configure your DHCP server to respond to the host. Using ISC DHCP, the following two options are all that's necessary in addition to the usual host stuff:
next-server 1.2.3.4;
filename "/debian/squeeze/amd64/pxelinux.0";

Where "next-server" is the address of your TFTP server and "filename" is the path to netboot's pxelinux.0 from the root of the tftpboot directory.

5. Boot to the installer

Your BMC should now be responding on the network. Use IPMItool to manipulate it.

You can use the -P <password> argument to ipmitool to avoid typing your password so much, but this will expose it to anyone who views the arguments (like via ps(1)) while the command is executing.

  1. Set the serial port speeds:
% ipmitool -I lanplus -H bmc.example.org -U root -P hax sol set non-volatile-bit-rate 57.6
% ipmitool -I lanplus -H bmc.example.org -U root -P hax sol set volatile-bit-rate 57.6
  1. Instruct the host to try to boot via PXE:
% ipmitool -I lanplus -H bmc.example.org -U root -P hax chassis bootdev pxe
Set Boot Device to pxe
  1. Reboot:
% ipmitool -I lanplus -H bmc.example.org -U root -P hax chassis power on
Chassis Power Control: Reset
  1. Connect to the system's console:
% ipmitool -I lanplus -H bmc.example.org -U root sol activate
Password:
[SOL Session operational.  Use ~? for help]

Your console should begin to show the hardware POST.

  1. If all is well you should see the boot process, the DHCP server will respond to the PXE request, and the host will load the SYSLINUX loader. Hit Return to select the default installer you defined in pxelinux.cfg/default.

If the install gets messed up and you need to restart, but you've already written some stuff to the disk, the BIOS will no longer attempt to PXE boot by default. Repeat step 2 if you need to restart the installation.

6. Configure Debian

  1. Once booted into the new system, set GRUB_CMDLINE_LINUX and fix the port speed in GRUB_SERIAL_COMMAND in /etc/default/grub to get full console port redirection to the serial port. Don't forget to run update-grub:
# grep GRUB_CMDLINE_LINUX= /etc/default/grub
GRUB_CMDLINE_LINUX="console=ttyS0,57600n8"
# grep GRUB_SERIAL_COMMAND= /etc/default/grub
GRUB_SERIAL_COMMAND="serial --unit=0 --speed=57600 --word=8 --parity=no --stop=1"
# update-grub
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-2.6.30-2-amd64
Found initrd image: /boot/initrd.img-2.6.30-2-amd64
done
#
  1. Install firmware-bnx2 from non-free so you have drivers in the installed OS. It's a good idea to preseed it, if you're preseeding.
# apt-get install firmware-bnx2
# reboot
  1. ???
  2. PROFIT!!!