================================================================ IBM x3550 - Debian Squeeze Installation via IPMI Serial Over Lan ================================================================ --------------------- How to make it happen --------------------- :Author: Nate Coraor :Version: $Revision: 0.2 $ :Copyright: Creative Commons Attribution Non-Commercial Share Alike 0. Preface ========== This document is an attempt at step-by-step instructions for installing Debian Squeeze on an IBM x3550 using IPMI's Serial Over LAN console abilities. It also assumes you're going to install Debian via a PXE netboot. These instructions are valid for Debian's other boot methods, but since you're cool enough to use IPMI, you should be using netboot too. I have not found a way to get into the BIOS over IPMI SOL. It seems you should be able to use ipmitool thusly: :: % ipmitool -I lanplus -H bmc.example.org chassis bootdev bios However, the BMC rejects this command: :: Set Chassis Boot Parameter 5 failed: Invalid data field in request And since neither ipmitool nor ipmiconsole seem to have a way to send the appropriate escape sequences to hit ````, it seems you're SOL for getting the BIOS over SOL (lawl). You can force a PXE boot, though, which is explained later. 1. Get the BMC on the network ============================= To configure the Baseboard Management Controller (BMC), you're probably going to need a monitor and keyboard. This is because the BMC is disabled from the factory, and I say probably because (if you're really motivated) you could create a USB drive that boots Linux and enables it from within the OS. But we're going to assume you're not going to do that. The x3550's *net1* is a shared BMC/host port. Note that the BMC is available in all x3550s. There is a seperate (optional) Remote Supervisor Adapter that uses the dedicated SM port. This document does not cover usage of the RSA and SM port. 2. BIOS/BMC Configuration ========================= 1. Boot up and enter BIOS setup: When the hardware boots, you will be presented with the opportunity to enter BIOS Setup: :: IBM BIOS - (c) Copyright IBM Corporation 2007 CP: 34 Symmetric Multiprocessing System Intel(R) Xeon(R) CPU X5460 @ 3.16GHz 2 Processor Packages 32768 MB Installed Memory Press F1 for Setup Press F2 for Diagnostics Press F12 to select boot device >> BIOS Version 1.07 << You'll want to strike ```` with gusto until the screen clears and ``Booting to Setup.`` appears. 2. Enable Remote Console Redirection: When the BIOS configuration loads, select ``Devices and I/O Ports``. Set ``Serial Port A`` to ``Auto configure``: :: _ ******************************************** * Devices and I/O Ports * ******************************************** * Serial Port A [ Auto*configure ] * * Serial Port B [ Disabled ] * * Remote Console Redirection * * * * Planar Ethernet 1 [ Enabled ] * * Planar Ethernet 2 [ Enabled ] * * Planar SAS [ Enabled ] * * PCI Slot 1 [ Enabled ] * * PCI Slot 2 [ Enabled ] * * * * * * Video * * IDE Configuration Menu * * System MAC Addresses * ******************************************** Next, select ``Remote Console Redirection`` and set as such: :: _ ******************************************** * Devices and I/O Ports * ******************************************************** * Remote Console Redirection * ******************************************************** * Remote Console Active [ Enabled ] * * Remote Console COM Port [ COM 1 ] * * Remote Console Baud Rate [ 57600 ] * * Remote Console Data Bits [ 8 ] * * Remote Console Parity [ None ] * * Remote Console Stop Bits [ 1 ] * * Remote Console Text Emulation [ VT100/VT220 ] * * Remote Console Keyboard Emulation [ VT100/VT220 ] * * Remote Console Active After Boot [ Enabled ] * * Remote Console Flow Control [ Disabled ] * ******************************************************** * System MAC Addresses * ******************************************** You may want to leave the ``Text Emulation`` and ``Keyboard Emulation`` options as ``ANSI``, which, if you're lucky, will draw nice ANSI pipe boxes instead of these ugly asterisks. Oddly enough this works fine for me on Dell PE1950s, but never draws correctly on the x3550. YMMV. Back out to the main BIOS screen by striking ```` with hasty abandon. 3. Assuming you're going to PXE boot, enable PXE on Planar Ethernet 1: Select ``Start Options``. The x3550 doesn't support 802.1q VLAN Tagging on the management interface, so you're going to have to either use the 2nd interface for the OS, or put both the management and OS on the same subnet. Choose Planar Ethernet 1 or 2 accordingly, here: :: _ ************************************************************ * Start Options * ************************************************************ * Startup Sequence Options * * * * Planar Ethernet PXE/DHCP [ Planar Ethernet 1 ] * * PCI Device Boot Priority [ Planar SAS ] * * Keyboard NumLock State [ Off ] * * USB Disk [ Enabled ] * * Boot on POST/BIOS Error [ Enabled ] * * Boot Fail Count [ Enabled ] * * Rehook INT 19h [ Disabled ] * * Virus Detection [ Disabled ] * ************************************************************ Back out to the main BIOS screen with an ````. 4. Configure the BMC Select ``Advanced Setup``, then select ``Baseboard Management Controller (BMC) Settings``. It should look like this: :: _ ********************************************************** * Baseboard Management Controller (BMC) Settings * ********************************************************** * BMC Firmware Version 01.34 * * BMC Build Level GFBT50A * * BMC Build Date 11*21*07 * * * * BMC POST Watchdog [ Disabled ] * * BMC POST Watchdog Timeout [ 5 Minutes ] * * System*BMC Serial Port Sharing [ Enabled ] * * BMC Serial Port Access Mode [ Shared ] * * Reboot System on NMI [ Enabled ] * * Power Restore Policy [ Always On ] * * BMC Fan Acoustic Mode [ Standard ] * * BMC Network Configuration * * BMC System Event Log * * User Account Settings * ********************************************************** Select ``BMC Network Configuration`` and configure it for your network. Examples, of course, your configuration will be different. :: _ ********************************************************** * Ba******************************************** * ********* BMC Network Configuration ******* * BMC ******************************************** * * BMC * BMC MAC Address 08*00*20*C0*FF*EE * * * BMC * Host Name [ 8020C0FFE ] * * * * DHCP Control [ Use Static IP ] * * * BMC * * * * BMC * IP Settings * * * Syst* IP Address [ 010.001.001.016 ] * * * BMC * Subnet Mask [ 255.255.255.000 ] * * * Rebo* Gateway [ 010.001.001.001 ] * * * Powe* * ] * * BMC * * * * BMC * Save Network Settings in BMC * * * BMC ******************************************** * * User Account Settings * ********************************************************** Then select ``Save Network Settings in BMC``. 5. Create an IPMI user: Select ``User Acccount Settings``: :: _ ********************************************************** * Baseboard Management Controller (BMC) Settings * ********************************************************** * BMC Firmware Version 01.34 * * **************************************************** * * * User Account Settings * * * **************************************************** * * * UserID# Username Enabled/Disabled * * * * UserID 1 NULL Disabled * * * * UserID 2 root Enabled * * * * UserID 3 UNUSED Disabled * * * * UserID 4 UNUSED Disabled * * * **************************************************** * * BMC Fan Acoustic Mode [ Standard ] * * BMC Network Configuration * * BMC System Event Log * * User Account Settings * ********************************************************** Disable ``NULL`` and create a new user with ``Administrator`` privileges. You can figure that out without me pasting screens, right? Make sure you select ``<>`` rather than just backing out with ````. Once users are configured, ```` out ot the main BIOS menu. 6. Save and exit: Select ``Save Settings`` to pop up the save dialog, and dismiss it. Strike ```` to pop up the Exit menu, and select ``Yes ...``. The host reboots. 4. Set up DHCP and netboot ========================== netboot is the image used to PXE boot and install Debian. Note that net'*inst*' is not net'*boot*'! 1. Set up a TFTP__ server somewhere. 2. Set up a DHCP__ server somewhere. Doesn't have to be the same physical server as TFTP. 3. Download ``netboot.tar.gz`` from the `testing Debian Installer page`__ and unpack it in (probably a subdirectory of) the TFTP server's directory (often ``/tftpboot`` or on Debian, ``/var/lib/tftpboot``). 4. Let's assume I stuck netboot in ``/tftpboot/debian/squeeze/amd64``. Modify the pxelinux config in ``pxelinux.cfg/default``, I generally just wipe out the contents and put what I'm actually going to install in there: :: default menu.c32 prompt 0 menu title PXE Boot Menu label squeeze-eth0-ttys0 menu label ^Squeeze via eth0/ttyS0 kernel debian/squeeze/amd64/linux append locale=en_US.UTF-8 interface=eth0 console-tools/archs=skip-config console-keymaps-at/keymap=us hostname=x domain=x preseed/url=http://foo.bar.baz/preseed-squeeze.cfg console=ttyS0,57600n8 DEBIAN_FRONTEND=text initrd=debian/squeeze/amd64/initrd.gz -- quiet DEBIAN_FRONTEND is optional. You may have success with the default curses interface, but if you have all kinds of screen draw problems, switch to text. The IPMI SOL console seriously blows, and especially the implementation on the x3550 (maybe I should update the BMC firmware?). 5. Add bnx2 firmware to the initrd image as explained here__. Otherwise, your installer won't be able to use the network devices. And that's kinda useless. 6. Configure your DHCP server to respond to the host. Using `ISC DHCP`__, the following two options are all that's necessary in addition to the usual host stuff: :: next-server 1.2.3.4; filename "/debian/squeeze/amd64/pxelinux.0"; Where "next-server" is the address of your TFTP server and "filename" is the path to netboot's ``pxelinux.0`` from the root of the ``tftpboot`` directory. .. __: http://en.wikipedia.org/wiki/Trivial_File_Transfer_Protocol .. __: http://en.wikipedia.org/wiki/Dynamic_Host_Configuration_Protocol .. __: http://www.debian.org/devel/debian-installer/ .. __: http://www.ducea.com/2009/03/02/debian-lenny-pxe-installation-on-dell-poweredge-19502950-servers-bnx2-annoyances/ .. __: https://www.isc.org/software/dhcp 5. Boot to the installer ======================== Your BMC should now be responding on the network. Use IPMItool__ to manipulate it. You can use the ``-P `` argument to ``ipmitool`` to avoid typing your password so much, but this will expose it to anyone who views the arguments (like via ``ps(1)``) while the command is executing. .. __: http://ipmitool.sourceforge.net/ 1. Set the serial port speeds: :: % ipmitool -I lanplus -H bmc.example.org -U root -P hax sol set non-volatile-bit-rate 57.6 % ipmitool -I lanplus -H bmc.example.org -U root -P hax sol set volatile-bit-rate 57.6 2. Instruct the host to try to boot via PXE: :: % ipmitool -I lanplus -H bmc.example.org -U root -P hax chassis bootdev pxe Set Boot Device to pxe 3. Reboot: :: % ipmitool -I lanplus -H bmc.example.org -U root -P hax chassis power on Chassis Power Control: Reset 4. Connect to the system's console: :: % ipmitool -I lanplus -H bmc.example.org -U root sol activate Password: [SOL Session operational. Use ~? for help] Your console should begin to show the hardware POST. 5. If all is well you should see the boot process, the DHCP server will respond to the PXE request, and the host will load the SYSLINUX loader. Hit ``Return`` to select the default installer you defined in ``pxelinux.cfg/default``. If the install gets messed up and you need to restart, but you've already written some stuff to the disk, the BIOS will no longer attempt to PXE boot by default. Repeat step 2 if you need to restart the installation. 6. Configure Debian =================== 1. Once booted into the new system, set ``GRUB_CMDLINE_LINUX`` and fix the port speed in ``GRUB_SERIAL_COMMAND`` in ``/etc/default/grub`` to get full console port redirection to the serial port. Don't forget to run ``update-grub``: :: # grep GRUB_CMDLINE_LINUX= /etc/default/grub GRUB_CMDLINE_LINUX="console=ttyS0,57600n8" # grep GRUB_SERIAL_COMMAND= /etc/default/grub GRUB_SERIAL_COMMAND="serial --unit=0 --speed=57600 --word=8 --parity=no --stop=1" # update-grub Generating grub.cfg ... Found linux image: /boot/vmlinuz-2.6.30-2-amd64 Found initrd image: /boot/initrd.img-2.6.30-2-amd64 done # 2. Install firmware-bnx2 from non-free so you have drivers in the installed OS. It's a good idea to preseed it, if you're preseeding. :: # apt-get install firmware-bnx2 # reboot 2. ??? 3. PROFIT!!!