Configuration of Host Hardware and OS for Coordinating Nodes

This document describes the setup of a virtual machine host on which will operate a DataONE Coordinating Node. The steps take for setup and configuration up to the point where a clean Ubuntu client is installed are described.

Hardware

Specifications of the hardware being installed for the UNM node.
Attribute Value
Brand, Model Dell, R905
Processors 4x Quad Core Opteron 8378 2.4GHz (Shanghai), 4x512K L2 Cache, 6MB L3 Cache, HyperTransport 1Ghz
Memory 64GB 800MHZ ECC (16X4GB)
Disks 5x 600GB 10K RPM Serial-Attach SCSI 3Gbps 3.5-in HotPlug HardDrive
Disk Controller PERC6i SAS RAID Controller Internal with Battery
NICs 4x Broadcom NetXtreme II 5708 1GbE Onboard NICs with TOE. Configured with LAG enabled.
Specifications of the hardware being installed for the UCSB node.
Attribute Value
Brand, Model Dell, R905
Processors 4x Quad Core Opteron 8380 2.5GHz (Shanghai), 4x512K L2 Cache, 6MB L3 Cache, HyperTransport 1Ghz
Memory 64GB 800MHZ ECC (16X4GB)
Disks 5x 600GB 15K RPM Serial-Attach SCSI 3Gbps 3.5-in HotPlug HardDrive
Disk Controller PERC6i SAS RAID Controller Internal with Battery
NICs 4x Broadcom NetXtreme II 5708 1GbE Onboard NICs with TOE. Configured with LAG enabled.

Disk Configuration

Disks are configured as RAID 1+0 (mirror + striping) with one hot spare, providing 1.2TB.

OS Installation

Boot from Ubuntu 9.10 64 bit server distribution.

Note

The OS was upgraded to 10.04 on 2010-05-10 with no apparent side effects besides the need to set the correct host name (reverted to “ubuntu”)

OS installation configuration values at UNM
Attribute Value
OS Ubuntu 10.04 64 bit server
Server name host-unm-1
Server FQDN host-unm-1.dataone.org
IP Address 129.24.0.18
Netmask 255.255.255.0
Gateway 129.24.0.1
DNS 1 129.24.0.52
DNS 2  
Partitions swap 16GB LVM everything else.
OS installation configuration values at UCSB
Attribute Value
OS Ubuntu 10.04 64 bit server
Server name host-ucsb-1
Server FQDN host-ucsb-1.dataone.org
IP Address 128.111.220.52
Netmask 255.255.255.0
Gateway 128.111.220.1
DNS 1 128.111.220.18
DNS 2 128.111.220.16
Partitions /boot 240MB LVM everything else.
OS installation configuration values at ORC
Attribute Value
OS Ubuntu 10.04 64 bit server
Server name orcvm1
Server FQDN orcvm1.dataone.org. host-orc-1.dataone.org
IP Address 160.36.134.68
Netmask 255.255.255.240
Gateway 160.36.234.65
DNS 1 160.36.134.65
Partitions /boot 240MB LVM everything else.

Partition setup at UCSB

  • during install, select “Guided - use entire disk and setup LVM”, then enter “40 GB” when asked how much space to use

  • after install extend swap and make partition for kvm:

    $ sudo lvextend /dev/host-ucsb-1/swap_1 -L 16G
$ sudo lvcreate -l 100%FREE -n kvm host-ucsb-1
$ sudo swapoff -a
$ sudo mkswap /dev/host-ucsb-1/swap_1
$ sudo swapon -a
$ sudo mkfs.ext4 /dev/host-ucsb-1/kvm

  • kvm partition mounted at the default ubuntu location /var/lib/libvirt/images and symlinked to /kvm, so as to not need to change /etc/libvirt config files

The partitions (UNM):

# parted -l
Model: DELL PERC 6/i Adapter (scsi)
Disk /dev/sda: 1199GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type      File system  Flags
 1      32.3kB  1199GB  1199GB  primary                boot, lvm
 2      1199GB  1199GB  255MB   extended
 5      1199GB  1199GB  255MB   logical   ext2


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/dataone-kvm: 1103GB
Sector size (logical/physical): 512B/512B
Partition Table: loop

Number  Start  End     Size    File system  Flags
 1      0.00B  1103GB  1103GB  ext4


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/dataone-home: 40.0GB
Sector size (logical/physical): 512B/512B
Partition Table: loop

Number  Start  End     Size    File system  Flags
 1      0.00B  40.0GB  40.0GB  ext4


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/dataone-root: 40.0GB
Sector size (logical/physical): 512B/512B
Partition Table: loop

Number  Start  End     Size    File system  Flags
 1      0.00B  40.0GB  40.0GB  ext4


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/dataone-swap: 16.0GB
Sector size (logical/physical): 512B/512B
Partition Table: loop

Number  Start  End     Size    File system     Flags
 1      0.00B  16.0GB  16.0GB  linux-swap(v1)

After the OS installation, setup for running KVM guests. Detailed instructions are available at the Ubuntu site

$ sudo apt-get install kvm libvirt-bin ubuntu-vm-builder bridge-utils
$ sudo adduser `id -un` libvirtd

Edit /etc/network/interfaces to support bridging so the KVM guests can use external IP addresses:

$ cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet manual

# The bridge network for KVM
auto br0
iface br0 inet static
      address 129.24.0.18
      netmask 255.255.255.0
      network 129.24.0.0
      broadcast 129.24.0.255
      gateway 129.24.0.1
        bridge_ports eth0
        bridge_stp off
        bridge_fd 0
        bridge_maxwait 0

      # dns-* options are implemented by the resolvconf package, if installed
      dns-nameservers 129.24.0.52
      dns-search nmepscor.net

Modify the default configuration for ufw to enable forwarding by editing /etc/default/ufw and set:

DEFAULT_FORWARD_POLICY="ACCEPT"

Then enable ssh and start the firewall:

sudo ufw allow ssh
sudo ufw enable

Network config at UCSB with two LACP bonded ports on an LACP enabled switch with bridging:

$ cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto bond0
iface bond0 inet manual
        bond-slaves none
        bond-mode 4
        bond-miimon 100

auto eth0
iface eth0 inet manual
        bond-master bond0
        bond-primary eth0 eth1

auto eth1
iface eth1 inet manual
        bond-master bond0
        bond-primary eth0 eth1

auto br0
iface br0 inet static
        address 128.111.220.52
        netmask 255.255.255.0
        gateway 128.111.220.1
        broadcast 128.111.220.255
        bridge_ports bond0
        bridge_fd 0
        bridge_maxwait 0

This material is based upon work supported by the National Science Foundation under Grant Numbers 083094 and 1430508.

Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.