UCSB KVM Host Setup

KVM Host Descriptions

UCSB has five physical servers hosting VMs named host-ucsb-1 to host-ucsb-5.

Host-ucsb-1 is a Dell server located at NCEAS, has about 2TB of local SAS storage.

Host-ucsb-2 to host-ucsb-5 are HP servers located on campus, the rest of this documentation is about them.

CentOS Configuration

CentOS 6 is installed on the HP servers because of hardware driver issues with the Emulex NICs and the current Ubuntu 10.04 LTS release. Setup instructions are below.

Install CentOS using the CentOS x86-64 DVD (6.2 at the time of writing) with the default install settings, except:

hostname "host-ucsb-#"
"Use All Space" automatic partition layout
"Virtual Host" software selection

Add users:

useradd username

Enable sudo for users (uncomment wheel, add users to wheel group):

visudo

Setup networking (bond0, br0, eth0, eth4, eth5, eth6, eth7):

(network config to be linked from here, currently in /etc/sysconfig/network-scripts

Setup /etc/resolv.conf:

vim /etc/resolv.conf
Add email alias for root to /etc/aliases::
vim /etc/aliases newaliases

Download and install epel repo:

wget http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-5.noarch.rpm

Install openssh-askpass, xorg-x11-xauth, device-mapper-multipath, etckeeper, fail2ban:

sudo yum update
sudo yum install xorg-x11-xauth device-mapper-multipath etckeeper fail2ban openssh-askpass

Init etckeeper, fail2ban:

sudo /etc/init.d/fail2ban start
sudo /sbin/chkconfig --level 2345 fail2ban on
sudo etckeeper init
sudo etckeeper commit "initial commit"

Create /etc/multipath.conf with the following contents:

defaults {
    user_friendly_names no
}

Start multipathd, set multipathd to start on boot:

sudo /etc/init.d/multipathd start
sudo /sbin/chkconfig --level 2345 multipathd on

Map iscsi devices:

sudo iscsiadm -m discovery -t sendtargets -p 10.0.1.10
sudo iscsiadm -m discovery -t sendtargets -p 10.0.1.11
sudo iscsiadm -m discovery -t sendtargets -p 10.0.1.12
sudo iscsiadm -m discovery -t sendtargets -p 10.0.1.13
sudo iscsiadm -m discovery -t sendtargets -p 10.0.2.10
sudo iscsiadm -m discovery -t sendtargets -p 10.0.2.11
sudo iscsiadm -m discovery -t sendtargets -p 10.0.2.12
sudo iscsiadm -m discovery -t sendtargets -p 10.0.2.13
sudo iscsiadm --mode node --targetname iqn.1986-03.com.hp:storage.p2000g3.1126130966 --portal 10.0.1.10:3260 --login
sudo iscsiadm --mode node --targetname iqn.1986-03.com.hp:storage.p2000g3.1126130966 --portal 10.0.2.10:3260 --login
sudo iscsiadm --mode node --targetname iqn.1986-03.com.hp:storage.p2000g3.112512ed28 --portal 10.0.1.11:3260 --login
sudo iscsiadm --mode node --targetname iqn.1986-03.com.hp:storage.p2000g3.112512ed28 --portal 10.0.2.11:3260 --login
sudo iscsiadm --mode node --targetname iqn.1986-03.com.hp:storage.p2000g3.1126130985 --portal 10.0.1.12:3260 --login
sudo iscsiadm --mode node --targetname iqn.1986-03.com.hp:storage.p2000g3.1126130985 --portal 10.0.2.12:3260 --login
sudo iscsiadm --mode node --targetname iqn.1986-03.com.hp:storage.p2000g3.11261308c5 --portal 10.0.1.13:3260 --login
sudo iscsiadm --mode node --targetname iqn.1986-03.com.hp:storage.p2000g3.11261308c5 --portal 10.0.2.13:3260 --login

Open ports for live migration in iptables:

sudo iptables -I INPUT -p tcp --dport 49152:49261 -j ACCEPT
sudo /etc/init.d/iptables save

Add public IP to /etc/hosts:

128.111.36.72   host-ucsb-2 host-ucsb-2.dataone.org

Add the following line to ~/.bashrc to run virt-manager over X:

alias virt-manager="XAUTHORITY=~/.Xauthority sudo virt-manager"

Add user to kvm group:

sudo vigr

Create and copy root ssh keys to all shared servers:

(run on each server)
sudo ssh-keygen
sudo ssh-copy-id host-ucsb-* (do this for all three other servers)
virt-manager
File > Add Connection > Connect to Remote host (SSH, root, host-ucsb-3, autoconnect) (do this for all three other servers)

Create a New Virtual Server

Create Storage Devices

Adding an iSCSI device:

Port-forward array management interface to your localhost, through host-ucsb-{2..5}:

ssh -N -f -L 8080:10.0.0.10:80 host-ucsb-2.dataone.org
ssh -N -f -L 8081:10.0.0.12:80 host-ucsb-2.dataone.org
ssh -N -f -L 8082:10.0.0.14:80 host-ucsb-2.dataone.org
ssh -N -f -L 8083:10.0.0.16:80 host-ucsb-2.dataone.org

Load management interface in a browser:

http://localhost:8080
http://localhost:8081
http://localhost:8082
http://localhost:8083

Login, create a new Volume on an existing Vdisk (with a unique LUN!) Make sure the LUN does not overlap at http://mule1.dataone.org/OperationDocs/ucsb-storage-layout.html

Rescan iSCSI devices on all four hosts:

sudo iscsiadm --mode node --targetname iqn.1986-03.com.hp:storage.p2000g3.1126130966 --portal 10.0.1.10:3260 --rescan
sudo iscsiadm --mode node --targetname iqn.1986-03.com.hp:storage.p2000g3.1126130966 --portal 10.0.2.10:3260 --rescan
sudo iscsiadm --mode node --targetname iqn.1986-03.com.hp:storage.p2000g3.112512ed28 --portal 10.0.1.11:3260 --rescan
sudo iscsiadm --mode node --targetname iqn.1986-03.com.hp:storage.p2000g3.112512ed28 --portal 10.0.2.11:3260 --rescan
sudo iscsiadm --mode node --targetname iqn.1986-03.com.hp:storage.p2000g3.1126130985 --portal 10.0.1.12:3260 --rescan
sudo iscsiadm --mode node --targetname iqn.1986-03.com.hp:storage.p2000g3.1126130985 --portal 10.0.2.12:3260 --rescan
sudo iscsiadm --mode node --targetname iqn.1986-03.com.hp:storage.p2000g3.11261308c5 --portal 10.0.1.13:3260 --rescan
sudo iscsiadm --mode node --targetname iqn.1986-03.com.hp:storage.p2000g3.11261308c5 --portal 10.0.2.13:3260 --rescan

Note the new UUID by getting the sd* names from dmesg, then matching them to multipath:

dmesg
multipath -ll

Record the UUID at http://mule1.dataone.org/OperationDocs/ucsb-storage-layout.html

Create the Virtual Server with virt-manager

Open virt-manager (can be run from any server):

XAUTHORITY=~/.Xauthority sudo virt-manager
  • Click New
  • Enter the name, connection, and choos Local install media
  • Select “Use ISO image:”, select the Ubuntu server ISO, choose OS type Linux, and version Ubuntu 10.04 to match the ISO
  • Assign memory and CPUs
  • choose “Select managed or other existing storage”, enter root multipath device (ex. /dev/mapper/3600c0ff0001274698df31d4f01000000)
  • check “Customize configuration before install”, then click Finish
  • remove the sound device
  • under Display VNC, choose Keymap en-us (default has issues with OS X)
  • add remaining multipath devices
    • click Add Hardware
    • choose Storage
    • Select managed or other existing storage
    • enter multipath device (ex. /dev/mapper/3600c0ff0001274698df31d4f01000000)
    • choose Device type “Virtio Disk”
    • click Finish
    • repeat
  • verify network device is bridged, and using the virtio device
  • click “Begin Installation”
  • choose “Install Ubuntu Server”
  • select defaults until the network config screen, DHCP config will fail, select Continue, then Configure Manually
  • assign a free IP address from the IP list and configure the network (doc page not yet available)
  • Partition the drives
    • for the initial config, select “Guided - use entire disk and set up LVM”
    • If there is only one virtual disk, you’re done. If there’s more than one, have the installer automatically setup the root device, then choose “No” when asked to write changes to the disk
    • configure the remaining disks, with LVM partitions and ext4 filesystems
  • select “openssh server” when prompted
  • finish the install using the defaults
  • reboot, virtual server install is now complete

Delete a Virtual Server

Removing iSCSI devices

  1. Follow the instructions, starting with #4, at http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/removing_devices.html
  • Each iSCSI device will have one /dev/mapper/, and two /dev/sd devices because of multipathing

Example for removing one iSCSI device from one host:

$ sudo multipath -ll 3600c0ff0001287547fa77c4f01000000
3600c0ff0001287547fa77c4f01000000 dm-45 HP,P2000 G3 iSCSI
size=186G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=0 status=active
| `- 9:0:0:24  sdxx 67:32  failed faulty running
`-+- policy='round-robin 0' prio=0 status=enabled
  `- 8:0:0:24  sdxy 67:0   failed faulty running
$ sudo multipath -f /dev/mapper/3600c0ff0001287547fa77c4f01000000
$ sudo blockdev --flushbufs /dev/sdxx
$ sudo blockdev --flushbufs /dev/sdxy
root# echo 1 > /sys/block/sdxx/device/delete
root# echo 1 > /sys/block/sdxy/device/delete
  1. Repeat for the other three hosts, note that /dev/sd* names will probably be different
  2. Delete the device from the P2000
  3. Update http://mule1.dataone.org/OperationDocs/ucsb-storage-layout.html

Resize a Virtual Server Disk Image

Resize the Disk

  1. Login to the web interface of the SAN and expand the disk image

  2. Find the devices for the disk image (do this on every iscsi host):

    sudo multipath -ll 3600c0ff0001287547fa77c4f01000000

  3. Refresh the size for the block device (do this on every iscsi host, note: the block devices will use different names!):

    $ echo 1 > /sys/block/sdaa/device/rescan $ echo 1 > /sys/block/sdab/device/rescan

  4. Refresh the size for the multipath device (do this on every iscsi host):

    $ sudo multipathd -k multipathd> resize map 3600c0ff0001287547fa77c4f01000000 ok multipathd> quit

  5. Verify the size has changed.

  6. Shut down (full off) and restart the virtual server to detect the changes

  7. Resize the partitions in the virtual servers

  • boot the virtual server from a GParted LiveCD ISO image
  • resize the partition holding the LVM VG to the new disk image size with the graphical partitioner
  • boot back into the virtual server
  • reize the LVM LV with lvextend
  • reize the file system with resize2fs

Live Migration

Live migration via virsh:

sudo virsh migrate --live --persistent --verbose cn-dev-ucsb-1 qemu+ssh://host-ucsb-4.dataone.org/system

Renaming Hosts

  • change /etc/hostname and /etc/hosts
  • change hostname with virt-manager or virsh
  • change the hostname of the storage devices (disk image files or iSCSI LUNs)
  • update DNS
  • update docs

Other Notes

  • iSCSI
    • each iscsi devices MUST have a unique LUN, across all four iSCSI arrays, or else CentOS won’t discover the new block devices without rebooting
  • IO-SRV
    • currently IO-SRV is available as a technology preview from Emulex. I did not enable it, opting for stability over speed. It will probably be stable soon.
    • IO-SRV needs iommu enabled, which I did not do, since it’s not currently needed
    • when IO-SRV is enabled in the NIC firmware, bonding breaks (I didn’t look into why)

This material is based upon work supported by the National Science Foundation under Grant Numbers 083094 and 1430508.

Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.