====================================== Coordinating Node Instance Upgrade ====================================== You can upgrade the software of a CN instance **only** if all DataONE ingest processes are stopped and inter-coordinating-node communications (for synchronizing datastores) are stopped. If you have not done this, see the Upgrade procedure. 1. Plan the Upgrade =================== There are several paths to upgrade a CN instance. Determine the following: - Will you need to backup the CN instance? - Is the upgrade limited to DataONE components? - Are there any OS upgrades also needed? - (Testing only): are you modifying any content collections? 2. Clone or Snapshot the CN =========================== Cloning procedures will be different on UCSB coordinating nodes than on UNM and ORC nodes. UCSB coordinating nodes use KVM for virtual machine technology while UNM and ORC use VMWare. This step is ignored for Staging. Cloning of Staging CN nodes will take place after staging has been used for final testing and any test data has been purged. In cases where only Metacat is being upgraded, the VM or LUN cloning can be skipped, but the Metacat database and on-disk files should be backed up according to a Metacat backup procedure. .. _`Metacat backup procedure`: ./metacat-backup-procedure.html 2.A KVM Procedure: SAN snapshots at the LUN level Procedure ------------------------------------------------------------ Login and shutdown virtual server (ex. cn-stage-ucsb-1) :: $ ssh cn-stage-ucsb-1.test.dataone.org $ sudo shutdown -h now Login to SAN, create LUN snapshot, verify snapshots :: $ ssh manage@10.0.0.xxx #. create snapshots volumes cn-stage-1-boot,cn-stage-ucsb-1 cn-stage-1-boot-snap,cn-stage-ucsb-1-snap #. show snapshots Boot virtual server :: $ ssh host-ucsb-x.dataone.org $ sudo virsh start cn-stage-ucsb-1 2.B VMWare Cloning Procedure ----------------------------- TODO: Enter in Dave’s Notes from Unm 3. Upgrade the Debian Packages ============================== 2.1 List the packages that will be upgraded --------------------------------------------- First ensure that the environment is using the correct release channel. The three channels are: ubuntu-unstable (for trunk), ubuntu-beta (branches), ubuntu-stable (tags). :: $ grep dataone /etc/apt/sources.list.d/dataone.list deb [arch=amd64] http://jenkins-1.dataone.org/ubuntu-beta precise universe deb [arch=amd64] http://jenkins-1.dataone.org/ubuntu-extra precise universe Note that in the example above, the CN is using the beta channel. The determine which packages will be upgraded :: $ sudo apt-get update $ sudo apt-get --simulate upgrade $ sudo apt-get --simulate dist-upgrade - if kernal patches are being upgraded, a VM reboot will be necessary - if Java is upgraded, then d1-cn-os-core will need to be reconfigured - to limit what is being upgraded, run the upgrade command with the name of the packages listed that you wish to upgrade. 2.2 Run the upgrade option -------------------------- :: $ sudo apt-get upgrade [package1 package2 ...] # OR $ sudo apt-get dist-upgrade [package1 package2 ...] # OR $ sudo apt-get --purge remove [package1 package2 ...] $ sudo apt-get install [package1 package2 ...] Depending on the packages being upgraded, follow up actions may be needed. - if Java7 is upgraded, then d1-cn-os-core will be need to be reconfigured :: $ sudo dpkg-reconfigure dataone-cn-os-core asks for Java7 keystore password asks for LDAP DB password - if there are kernal patches, a reboot of the VM will be necessary :: $ sudo reboot 2.3 Confirm Base services are running ------------------------------------- Confirm that Tomcat7 is running :: $ tail -500f /var/log/tomcat7/catalina.out | grep -P '(org.apache.catalina.startup.Catalina start)|(INFO: Server startup in)' Confirm that OpenLDAP is running, if not, then restart :: $pidof slapd The pid of slapd will be returned if it is running. To restart: :: $ sudo service slapd start Confirm that Postgres is running, if not, then restart: :: $pidof postgres The pids of postgres will be returned if it is running. To restart: :: $ sudo service postgres start 3. Re-Configure Metacat ======================== If Metacat has been upgraded, use the web admin interface to re-configure Metacat: Navigate to: https://$CN1|$CN2|$CN3.dataone.org/metacat/admin Once at the Admin site, you will first be taken to the Authentication Configuration page. Enter the password from SystemPW.txt file for the Metacat Administrator fields. (The subject should start with: "uid=dataone_cn_metacat") You will need to visit the following pages in the Admin site after this to complete Metacat Configuration: - Metacat Global Properties: use the default values - Database Installation/Upgrade: do as told, unless you have different directions - Geoserver Configuration: bypass - DataONE Configratuion: bypass - this page is only for Metacat as a Member Node - Replication Configuration: check that entries for the other CN instances are listed in the bottom panel of this page. If not, add them in the servers section, using the server's DNS name, and setting Replicate Metadata? and Replicate Data? to 'Yes', and Localhost is Hub to 'No' After metacat has been configured, restart tomcat :: $ sudo service tomcat7 stop $ sudo service tomcat7 start NOTE: It may take tomcat7 30-60 minutes to restart 4. Optionally Clear the Search Index ===================================== If you've just cleared Metacat, it's usually a good idea to clear the SOLR index. Otherwise you'll have indexed content for objects not yet registered to the DataONE environment! To do this, run the following script: :: /usr/share/dataone-cn-index/scripts/clearSearchIndex.sh 5. Optionally Rebuild and copy search index =========================================== Occasionally when new search index fields are added to the search index schema, a rebuild of the entire index is needed. This procedure should be followed on a CN that is currently out of round robin DNS and after the index rebuild is complete, the solr data files copied to the other 2 CN while they are out of round robin DNS. To rebuild search index with existing cn data (after an upgrade, if not clearing CN documents/data): :: $ sudo nohup /usr/share/dataone-cn-index/scripts/index-tool.sh & To Copy index data files from one CN to another (to avoid rebuilding on each CN) COPY INDEX FILES: (example shows moving index data files from unm to orc) :: $ ssh -A user@cn-stage-unm-1.dataone.org (from workstation to search index source machine, -A forwards credentials) ## stop tc and d1-index-task-generator, d1-index-task-processor (d1-processing if running) deamons on source CN (stops mutations of index data files) $ cp -r /var/lib/solr/data/d1-cn-index/index/ /home/user/indexCopy (indexCopy is a temp dir, can be removed after rsync) ## start tc and d1-index-task-generator, d1-index-task-processor, (d1-processing) daemons on source CN $ rsync -av --partial /home/user/indexCopy user@cn-stage-orc-1.dataone.org:/home/user/indexCopy (copy from source to target machine, over 4GB of files to move so may take a bit) ## stop tc and d1-index-task-generator, d1-index-task-processor (d1-processing if running) deamons on target CN (cn-stage-orc-1) $ rm -rf /var/lib/solr/data/d1-cn-index/index/* (remove old index data files) $ cp -r /home/user/indexCopy/* /var/lib/solr/data/d1-cn-index/index/ (copy new data files to solr data dir) $ sudo chown -R tomcat7.tomcat7 /var/lib/solr/data/d1-cn-index/index/ (assign data files to tomcat7 user/group) $ start tc, d1-index-task-generator, d1-index-task-processor (d1-processing) daemons on target CN (cn-stage-orc-1). Previous Steps Regarding Backups ================================ The Documentation below is the previous instructions for LVM snapshotting that did not work on Ubuntu 10.04. The instructions are saved for reference for after the upgrade to 12.04. 0.1 Create filesystem backups using LVM snapshots ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 0.2 Ensure there's enough space in the LVM Volume Group ------------------------------------------------------- The LVM volume group for a set of logical volumes needs enough space to accommodate: * The growth of each snapshot up to the full size of the original logical volume * A copy of the the snapshot logical volume (which will replace the current logical volume) * A minor amount for LVM metadata used to keep track of chunk locations (see lvcreate -s option) As an example, for a ``/var`` data partition of 1TB, we'll need a volume group with: * 1TB for the origin ``/var`` logical volume * 1TB for the ``/var-snapshot`` logical volume * 1TB for the copy of ``/var-snapshot`` * 1GB for LVM metadata (not sure about this, probably a conservative estimate) * **Total 3.1TB** The following table shows the current volume group sizes, logical volume sizes, and the needed additions for each CN in the stage and production environments (as of 12Jan2013). In the ``Required VG Size`` column, the first value is the minimum needed (2 x VG Size + Snapshot Size + 1G overhead), and the second value is the more liberal needed value ( 3 x VG Size + 1G overhead), as described in the example above. We need to discuss which of these values is most appropriate. The last column, a ``New VG Size`` of 4 TB would accommodate either scenario and would let us deal with filesystem growth. Likewise, for simplicity, the three UCSB volume groups could be consolidated into a single volume group like ORC and UNM. +---------------------+------------------+---------------------+----------------------------------+-------------+-----------------------+--------------+ | CN | VG Name | VG Size | LV Name | LV Size | Required VG Size | New VG Size | +=====================+==================+=====================+==================================+=============+=======================+==============+ | cn-stage-ucsb-1 | cn-ucsb-1 | 46.32 GB | root | 38.87 GB | 82.74 GB/117.61 GB | **4 TB** | + + + +----------------------------------+-------------+ + + | | | | snap-root-2012-12-07-1354857208 | 5.00 GB | | | + +------------------+---------------------+----------------------------------+-------------+-----------------------+ + | | cn-ucsb-1-usr | 93.13 GB | usr | 93.13 GB | 197.26 GB/280.39 GB | | + + + +----------------------------------+-------------+ + + | | | | snap-usr-2012-12-07-1354857208 | 10.00 GB | | | + +------------------+---------------------+----------------------------------+-------------+-----------------------+ + | | cn-ucsb-1-var | 931.32 GB | var | 931.32 GB | 1963.64 GB/2794.96 GB | | + + + +----------------------------------+-------------+ + + | | | | snap-var-2012-12-07-1354857208 | 100.00 GB | | | +---------------------+------------------+---------------------+----------------------------------+-------------+-----------------------+--------------+ | cn-stage-orc-1 | cn-orc-1-VG | 1140.00 GB | cn-orc-1-LV | 37.25 GB | 2396.00 GB/3421.00 GB | **4 TB** | + + + +----------------------------------+-------------+ + + | | | | snap-root-2012-12-07-1354857208 | 5.00 GB | | | + + + +----------------------------------+-------------+ + + | | | | cn-orc-1-LV2 | 93.13 GB | | | + + + +----------------------------------+-------------+ + + | | | | snap-usr-2012-12-07-1354857208 | 10.00 GB | | | + + + +----------------------------------+-------------+ + + | | | | cn-orc-1-LV3 | 931.32 GB | | | + + + +----------------------------------+-------------+ + + | | | | snap-var-2012-12-07-1354857208 | 100.00 GB | | | +---------------------+------------------+---------------------+----------------------------------+-------------+-----------------------+--------------+ | cn-stage-unm-1 | ubuntu | 1110.00 GB | root | 37.25 GB | 2336.00 GB/3331.00 GB | **4 TB** | + + + +----------------------------------+-------------+ + + | | | | snap-root-2012-12-07-1354857208 | 5.00 GB | | | + + + +----------------------------------+-------------+ + + | | | | usr | 93.13 GB | | | + + + +----------------------------------+-------------+ + + | | | | snap-usr-2012-12-07-1354857208 | 10.00 GB | | | + + + +----------------------------------+-------------+ + + | | | | var | 931.32 GB | | | + + + +----------------------------------+-------------+ + + | | | | snap-var-2012-12-07-1354857208 | 100.00 GB | | | +---------------------+------------------+---------------------+----------------------------------+-------------+-----------------------+--------------+ | cn-ucsb-1 | cn-ucsb-1 | 46.32 GB | root | 38.87 GB | 82.74 GB/117.61 GB | **4 TB** | + + + +----------------------------------+-------------+ + + | | | | snap-root-2013-01-1X-XXXXXXXXXX | 5.00 GB | | | + +------------------+---------------------+----------------------------------+-------------+-----------------------+ + | | cn-ucsb-1-usr | 93.13 GB | usr | 93.13 GB | 197.26 GB/280.39 GB | | + + + +----------------------------------+-------------+ + + | | | | snap-usr-2013-01-1X-XXXXXXXXXX | 10.00 GB | | | + +------------------+---------------------+----------------------------------+-------------+-----------------------+ + | | cn-ucsb-1-var | 931.32 GB | var | 931.32 GB | 1963.64 GB/2794.96 GB | | + + + +----------------------------------+-------------+ + + | | | | snap-var-2013-01-1X-XXXXXXXXXX | 100.00 GB | | | +---------------------+------------------+---------------------+----------------------------------+-------------+-----------------------+--------------+ | cn-orc-1 | cn-orc-1-VG | 1140.00 GB | cn-orc-1-LV | 37.25 GB | 2396.00 GB/3421.00 GB | **4 TB** | + + + +----------------------------------+-------------+ + + | | | | snap-root-2013-01-1X-XXXXXXXXXX | 5.00 GB | | | + + + +----------------------------------+-------------+ + + | | | | cn-orc-1-LV2 | 93.13 GB | | | + + + +----------------------------------+-------------+ + + | | | | snap-usr-2013-01-1X-XXXXXXXXXX | 10.00 GB | | | + + + +----------------------------------+-------------+ + + | | | | cn-orc-1-LV3 | 931.32 GB | | | + + + +----------------------------------+-------------+ + + | | | | snap-var-2013-01-1X-XXXXXXXXXX | 100.00 GB | | | +---------------------+------------------+---------------------+----------------------------------+-------------+-----------------------+--------------+ | cn-unm-1 | ubuntu | 1110.00 GB | root | 37.25 GB | 2336.00 GB/3331.00 GB | **4 TB** | + + + +----------------------------------+-------------+ + + | | | | snap-root-2013-01-1X-XXXXXXXXXX | 5.00 GB | | | + + + +----------------------------------+-------------+ + + | | | | usr | 93.13 GB | | | + + + +----------------------------------+-------------+ + + | | | | snap-usr-2013-01-1X-XXXXXXXXXX | 10.00 GB | | | + + + +----------------------------------+-------------+ + + | | | | var | 964.16 GB | | | + + + +----------------------------------+-------------+ + + | | | | snap-var-2013-01-1X-XXXXXXXXXX | 100.00 GB | | | +---------------------+------------------+---------------------+----------------------------------+-------------+-----------------------+--------------+ If there isn't enough space on a given VM for the above: a) Add new physical (or virtual) disks to the VM (contact Nick Brand, Jamin Ragle, Chris Brumgard) b) Rescan the SCSI bus to inform the kernel of the new device :: # echo "- - -" > /sys/class/scsi_host/host#/scan (Where # is the host scsi adapter [0,1,2,...]. Rescan all if need be.) c) For understanding LUN to kernel device name mapping, install lsscsi :: # apt-get install lsscsi (This should go into our Ansible config) # lsscsi [1:0:0:0] cd/dvd NECVMWar VMware IDE CDR10 1.00 /dev/sr0 [2:0:0:0] disk VMware Virtual disk 1.0 /dev/sda [2:0:1:0] disk VMware Virtual disk 1.0 /dev/sdb [2:0:2:0] disk VMware Virtual disk 1.0 /dev/sdc The newly added physical disk will likely be at /dev/sdb or /dev/sdc d) Create LVM partitions with the new disk (optional, can use whole disk too) :: # fdisk /dev/sdb In fdisk, choose: n (new partition) p (primary) 1 (choose partition size by sectors) t (set the partition type) 8e (Linux LVM) w (write the partition table) e) pvcreate /dev/sdb (or /dev/sdb1 if you created a partition) f) vgextend ubuntu /dev/sdb (Note that the VG on cn-stage-unm-1 is 'ubuntu'. Others vary) 0.3 Create the snapshot at the rollback instance before any software upgrade ---------------------------------------------------------------------------- Run the `LVM snapshot wrapper script`_: :: # nohup lvm-snapshot.sh > lvm-snapshot.out 2> lvm-snapshot.err < /dev/null & .. _LVM snapshot wrapper script: https://repository.dataone.org/software/cicore/trunk/cn/d1_cn_tools/d1_cn_snapshot/ 0.4 In the Stage environment, roll back to the previous stable release ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ During an upgrade release of the production environment, we first test the installations out in the stage environment. We'll snapshot the stage VMs, create a 1_X_BRANCH in subversion, install that branch in the stage environment, and test it. If testing succeeds, we create a 1_X_TAG in subversion, and will roll the stage environment back to it's stable state to emulate the production upgrade. The following procedures describe this rollback. 0.4.1 Backup the ``/boot`` filesystem ----------------------------------- The ``/boot`` filesytem is not an LVM volume, and it will be overwritten when the root logical volume is rolled back. As root, Back it up with ``dd``. :: root@cn-stage-unm-1:~# df /boot Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 233191 35961 184789 17% /boot # umount /boot # dd if=/dev/sda1 of=dev_sda1_backup (replace /dev/sda1 with /dev/vda1 for ucsb hosts) # mount /boot Once ``/boot`` is backed up, rsync the backup to another host to ensure it can be restored 0.4.2 Rename the logical volumes for ``/``, ``/usr``, and ``/var`` ---------------------------------------------------------------- By renaming the logical volumes, we can replace them with the snapshot versions taken earlier without booting off of external media. Note that UCSB and ORC volume groups are named differently, so change the rename commands appropriately. List the logical volume names and their logical extent sizes for later use. :: # lvdisplay | egrep "LV Name|Current LE" LV Name /dev/ubuntu/swap_1 Current LE 11576 LV Name /dev/ubuntu/root Current LE 9536 LV Name /dev/ubuntu/usr Current LE 23841 LV Name /dev/ubuntu/var Current LE 246825 LV Name /dev/ubuntu/snap-root-2012-12-07-1354857095 Current LE 9536 LV Name /dev/ubuntu/snap-usr-2012-12-07-1354857095 Current LE 23841 LV Name /dev/ubuntu/snap-var-2012-12-07-1354857095 Current LE 246825 # lvrename /dev/ubuntu/root /dev/ubuntu/root_old # lvrename /dev/ubuntu/usr /dev/ubuntu/usr_old # lvrename /dev/ubuntu/var /dev/ubuntu/var_old 0.4.3 Create the replacement logical volumes for the originals ------------------------------------------------------------ We'll be copying the snapshot instance to the logical volume names that were originally used, so create the same size volume as what was listed before. :: # lvcreate -l 9536 -n root /dev/ubuntu # lvcreate -l 23841 -n usr /dev/ubuntu # lvcreate -l 246825 -n var /dev/ubuntu 0.4.4 Push the snapshot contents into the new logical volumes ----------------------------------------------------------- We effectively copy the snapshot volume bytes into a new logical volume. We do this to avoid performance problems that have been documented with LVM snapshot volumes in comparison to non-snapshot volumes. `Some tests`_ show an order of magnitude decrease in write performance, so we're trying to avoid that. Not that block size will affect performance, and a block size of 1M is suggested based on transfer rates at UCSB. :: # dd if=/dev/ubuntu/snap-root-2012-12-07-1354857095 of=/dev/ubuntu/root bs=1M # dd if=/dev/ubuntu/snap-usr-2012-12-07-1354857095 of=/dev/ubuntu/usr bs=1M # dd if=/dev/ubuntu/snap-var-2012-12-07-1354857095 of=/dev/ubuntu/var bs=1M .. _Some tests: http://www.nikhef.nl/~dennisvd/lvmcrap.html 0.4.5 Restore the ``/boot`` filesystem -------------------------------- Copy the bytes of the ``/boot`` backup to it's original partition. :: # umount /boot # dd if=dev_sda1_backup of=/dev/sda1 (dev_vda1_backup and /dev/vda1 for UCSB hosts) # mount /boot 0.5. Reboot the VM ~~~~~~~~~~~~~~~~ Now that we've replaced the original logical volumes with the snapshot data, bring the VM up with the new volumes as active. :: # shutdown -r now 0.6. Remove unneeded logical volumes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To free up space in the volume group for this procedure to be done again, remove both the original logical volume and the snapshot :: # lvremove /dev/ubuntu/root_old # lvremove /dev/ubuntu/usr_old # lvremove /dev/ubuntu/var_old # lvremove /dev/ubuntu/snap-root-2012-12-07-1354857095 # lvremove /dev/ubuntu/snap-usr-2012-12-07-1354857095 # lvremove /dev/ubuntu/snap-var-2012-12-07-1354857095