Upgrading a Coordinating Node

In order to upgrade Coordinating Nodes (CN) without downtime, we follow a procedure that allows us to switch the CN in the DNS round robin. We disconnect two passive master Nodes from the cluster and database software in order ensure that we will not have different software versions that may cause data conflicts after an upgrade. We back up the filesystem partitions using LVM snapshots such that we have a consistent point in time to roll back to in the case of a major upgrade problem. We then upgrade the CN software to the newest release for the two machines, and then restore cluster and database communication between these machines. Once these machines are in a stable state, and all data stores have also been upgraded (for instance new index schema, new LDAP schema, new PostgreSQL schema), we switch the IP in the DNS round robin to one of the upgraded nodes simultaneously removing the active master CN.

The same procedure is then applied to the active master CN, and it is returned to the DNS round robin replacing one of the passive master CNs. This overall procedure is outlined in detail below, using CN1, CN2, and CN3 to represent the three CNs.

Before starting a new release process, send a release announcement to the operations listserve at DataONE to notify the community of the changes. Detail Wiki pages with Release Notes should be created for each release level on redmine.dataone.org, and a link to the wiki page sent along with the release announcement.

Current Production CNs

IP	FQDN	Tag
128.111.54.80	cn-ucsb-1.dataone.org	CN1
160.36.13.150	cn-orc-1.dataone.org	CN2
64.106.40.6	cn-unm-1.dataone.org	CN3

0. Clone or Snapshot All CNs

Cloning procedures will be different on UCSB coordinating nodes than on UNM and ORC nodes. UCSB coordinating nodes use KVM for virtual machine technology while UNM and ORC use VMWare. This step is ignored for Staging. Cloning of Staging CN nodes will take place after staging has been used for final testing and any test data has been purged. In cases where only Metacat is being upgraded, the VM or LUN cloning can be skipped, but the Metacat database and on-disk files should be backed up according to a Metacat backup procedure.

0.1 SAN snapshots at the LUN level Procedure

Login and shutdown virtual server (ex. cn-stage-ucsb-1)

$ ssh cn-stage-ucsb-1.test.dataone.org
$ sudo shutdown -h now

Login to SAN, create LUN snapshot, verify snapshots

$ ssh manage@10.0.0.xxx

# create snapshots volumes cn-stage-1-boot,cn-stage-ucsb-1 cn-stage-1-boot-snap,cn-stage-ucsb-1-snap # show snapshots

Boot virtual server

$ ssh host-ucsb-x.dataone.org
$ sudo virsh start cn-stage-ucsb-1

0.2 VMWare Cloning Procedure

Establish a VPN connection to UNM from https://vpn.unm.edu

Login to the UNM VPN site, and install the NetworkConnect desktop software for future VPN connections

Once the VPN connection is established, browse to https://libcon.unm.edu:9443/vsphere-client

Login to VSphere using your UNM credentials (e.g. collegesusername, password)

Once in the VSphere web client, navigate to the CN virtual machine that needs to be cloned. Clone the two passive CNs first (not in the round robin), by right-clicking on the VM name and walking through the ‘Clone to Virtual Machine ...’ dialogs. Choose a cloned VM name that reflects the date it was clone (like cn-unm-1-clone-20150901).

1. Remove two CNs from the DNS round robin

Originally we had three active CNs in a multimaster setup. As of 11/05/2014 (this edit), we have only a single Active Master CN with two passive Master Nodes.

The inactive CNs should already be removed from the DNS RR, but it is always good to confirm

1.1 DataONE v2.0 install/upgrade step

DataONE version 2.0 will include a few new software components to support the additon of solr cloud (solr 5.2.1) based search indexes. Before proceeding through the usual upgrade proceedure, these components need to be installed across the CN cluster - as this allows the solr cloud components to operate properly during installation of dataone search indexes. These installs do not effect/interact with current services on the CN machines so they can be installed during normal processing operations: On each CN:

sudo apt-get update
sudo apt-get install dataone-zookeeper dataone-solr

Once the installs complete a zookeeper quarum will be running across the CN cluster. A cloud based solr instance will also be running across the CN cluster - waiting for collections(cores) to be installed. To view and verify the zookeeper cluster quarum view the file zoo.cfg file installed at:

/var/lib/zookeeper/conf/zoo.cfg

In this file, each member of the cluster should be represented with a line beginning “server.x” and declaring the server’s IP and zookeeper server ports.

To verify zookeeper is running try connecting a client to each member of the CN zookeeper cluster use this command:

sudo /etc/init.d/zookeeper status

If running verify further with this command (use zoo.cfg to find the IP address for each cluster member as well as the zookeeper client port):

/var/lib/zookeeper/bin/zkCli.sh -server SERVER_IP:ZOOKEEPER_CLIENT_PORT

This script should connect to a zookeeper server and allow user to run simple commands like ‘ls’ to view the zookeeper file system.

To verify that solr cloud is running:

sudo /etc/init.d/solr status

Once zookeeper and solr are installed and running on each of the CN machines, deployment can continue.

2. Toggle Read-Only Mode for Active CN

Turn off the processing and indexing services

2.1 Turn off d1 processing on Active CN

Confirm that d1-processing is still running

root@cn-ucsb-1:/etc/dataone/process#  ps -p $(cat /var/run/d1-processing.pid)
PID TTY          TIME CMD
15411 ?        00:03:20 jsvc
root@cn-ucsb-1:/etc/dataone/process# echo $?
0

Set all the processing components to inactive. In /etc/dataone/process there are three property files:

• logAggregation.properties
• synchronization.properties
• replication.properties

In logAggregation.properties set the LogAggregator.active to FALSE. In synchronization.properties set the Synchronization.active to FALSE. In replication.properties set the Replication.active to FALSE.

Check each of the logging files for logAggregation, synchronization and replication to ensure that the processing components have been inactivated.

Stop Processing

$ sudo /etc/init.d/d1-processing stop

Note that this step is only for active CN in the cluster, currently cn-ucsb-1.dataone.org is the designated active CN

2.2 Turn off indexing services on all 3 CNs

Stop Generator and Processor

$ sudo /etc/init.d/d1-index-task-processor stop
$ sudo /etc/init.d/d1-index-task-generator stop

2.3 Toggle read-only and disconnect cluster communication on Active CN

This script will turn off the correct ports and perform other settings manipulation needed for the upgrade procedure. If it’s not installed in /usr/local/bin on the CNs, add it from https://repository.dataone.org/software/tools/trunk/control_release/togglePortsAndReplication.sh.

$ sudo /usr/local/bin/togglePortsAndReplication.sh disable

The /etc/dataone/node.properties file should have a property named ‘cn.storage.readOnly’, if not, add it The property should be set to TRUE on CN1, CN2 and CN3

3. Disconnect the passive CN’s cluster communication

Each of the CNs should be isolated from the others. On all passive nodes, run the command:

$ sudo /usr/local/bin/togglePortsAndReplication.sh disable

4. Upgrade the VMs to the next release of the CN software stack

Make certain you are a part of the sys admin group who has access to passwords.

4.1 Upgrade the software of the passive Master Nodes

Ensure that the environment is using the correct release channel. The three channels are: ubuntu-unstable (trunk), ubuntu-beta (branches), ubuntu-stable (tags).

$ grep dataone /etc/apt/sources.list.d/dataone.list
deb [arch=amd64] http://jenkins-1.dataone.org/ubuntu-beta precise universe
deb [arch=amd64] http://jenkins-1.dataone.org/ubuntu-extra precise universe

Note that in the example above, the CN is using the beta channel.

If there are kernel patches, use dist-upgrade below, and reboot the VM appropriately.

$ sudo apt-get update
$ sudo apt-get upgrade (or)
$ sudo apt-get dist-upgrade (if new server image is needed)

During apt-get upgrade if Java7 is upgraded, then d1-cn-os-core will need to be reconfigured.

$ sudo dpkg-reconfigure dataone-cn-os-core
asks for Java7 keystore password
asks for LDAP DB password

Add in the command to perform a reboot of a machine if necessary

$ sudo reboot

Re-enable Cluster communications:

$ sudo /usr/local/bin/togglePortsAndReplication.sh enable

4.1.1 Re-Configure Metacat

If Metacat has been upgraded, use the web admin interface to re-configure Metacat: Navigate to: https://$CN2|$CN3.dataone.org/metacat/admin

Once at the Admin site, you will first be taken to the Authentication Configuration page. Enter the password from SystemPW.txt file for the Metacat Administrator fields. (The subject should start with: “uid=dataone_cn_metacat”)

You will need to visit the following pages in the Admin site after this to complete Metacat Configuration:

• Metacat Global Properties: use the default values
• Database Installation/Upgrade: do as told, unless you have different directions
• Geoserver Configuration: bypass
• DataONE Configratuion: bypass - this page is only for Metacat as a Member Node

After metacat has been configured, restart tomcat

$ sudo service tomcat7 stop
$ sudo service tomcat7 start

NOTE: It may take tomcat7 30-60 minutes to restart

4.1.2 Optionally Clear the Search Index

If you’ve just cleared Metacat, it’s usually a good idea to clear the SOLR index. Otherwise you’ll have indexed content for objects not yet registered to the DataONE environment! To do this, run the following script:

/usr/share/dataone-cn-index/scripts/clearSearchIndex.sh

4.1.3 DataONE v2.0 install/upgrade steps (populate search core in solr cloud)

Since the DataONE v2.0 release includes the migration to solr cloud (solr 5.2.1), the search indexes must be re-created in the new solr install. This will occur on a single passive CN, however due to solr cloud operation (supported by zookeeper) the search index will actually be created on each member of the CN cluster. Assumes the following packages are installed:

dataone-cn-solr (v2.0) will create and configure the search and log event collections (cores) in solr cloud.
dataone-cn-index (v2.0) will install support in indexing to target the new solr cloud install.

While the first (or second passive) CN is still out of the DNS RR start a search index rebuild:

sudo nohup /usr/share/dataone-cn-index/scripts/index-tool.sh &

This process will take several days to complete. Monitor the nohup output file to determine when the process has completed. Once finished the new solr cloud search index will be available on all CN machines.

4.2 Change RR DNS so that CN2 acts as the active read-only CN in order to upgrade CN1

CN2 is usually one of the ORC boxes

4.3 Upgrade the software of the active Master Nodes

Ensure that the environment is using the correct release channel. The three channels are: ubuntu-unstable (trunk), ubuntu-beta (branches), ubuntu-stable (tags).

$ grep dataone /etc/apt/sources.list.d/dataone.list
deb [arch=amd64] http://jenkins-1.dataone.org/ubuntu-beta precise universe
deb [arch=amd64] http://jenkins-1.dataone.org/ubuntu-extra precise universe

Note that in the example above, the CN is using the beta channel.

Shutdown subsystems:

$ sudo /usr/local/bin/toggleDataONEService.sh disable

If there are kernel patches, use dist-upgrade below, and reboot the VM appropriately.

$ sudo apt-get update
$ sudo apt-get upgrade (or)
$ sudo apt-get dist-upgrade (if new server image is needed)

During apt-get upgrade if Java7 is upgraded, then d1-cn-os-core will need to be reconfigured.

$ sudo dpkg-reconfigure dataone-cn-os-core

asks for Java7 keystore password
asks for LDAP DB password

Add in the command to perform a reboot of a machine if necessary

$ sudo reboot

Confirm that Tomcat7 is running

$ tail -500f /var/log/tomcat7/catalina.out | grep -P '(org.apache.catalina.startup.Catalina start)|(INFO: Server startup in)'

Confirm that OpenLDAP is running, if not, then restart

$pidof slapd

The pid of slapd will be returned if it is running. To restart OpenLDAP...

$ sudo service slapd start

Confirm that Postgres is running, if not, then restart

$pidof postgres

The pids of postgres will be returned if it is running

To restart OpenLDAP

$ sudo service postgres start

If Metacat has been upgraded, use the web admin interface to re-configure Metacat:

Navigate to: https://$CN2|$CN3.dataone.org/metacat/admin

Use the Admin screens to configure metacat, after metacat has been configured, restart tomcat

sudo service tomcat7 stop
sudo service tomcat7 start

NOTE: It may take tomcat7 30-60 minutes to restart

TODO: Describe how to reconfigure metacat via web interface.

Re-enable Cluster communications:

$ sudo /usr/local/bin/togglePortsAndReplication.sh enable

4.4 Change RR DNS so that Active Master is only machine in RR

5. Restart D1-Processing and Indexing Daemons

Turn on the processing and indexing services

5.1 Turn on indexing services on a single CN

With DataONE v2.0, the index service daemons no longer need to run on each CN. Due to solr cloud operation mode, the indexing services only need to be started on a single CN. Ensure that indexing services are running only on a single CN: Start up Processor and Generator

$ sudo /etc/init.d/d1-index-task-processor start
$ sudo /etc/init.d/d1-index-task-generator start

The search index population command run in step 4.1.3 will start the indexing services on the CN where the script is run, so starting the services on a CN may not be necessary.

5.2 Turn on the processing on the Active Master Node

Set all the processing components to active.

In /etc/dataone/process there are three property files: * logAggregation.properties * synchronization.properties * replication.properties

In logAggregation.properties set the LogAggregator.active to TRUE. In synchronization.properties set the Synchronization.active to TRUE. In replication.properties set the Replication.active to TRUE.

Start up Processing

$ sudo /etc/init.d/d1-processing start

• Note that this step is only for active CN in the cluster, currently cn-ucsb-1.dataone.org is the designated active CN

6. Rebuild and copy search index (This step is deprecated and no long needed due to solr cloud operation with DataONE v2.0)

(This step is deprecated and no long needed due to solr cloud operation with DataONE v2.0)

Occasionally when new search index fields are added to the search index schema, a rebuild of the entire index is needed. This procedure should be followed on a CN that is currently out of round robin DNS and after the index rebuild is complete, the solr data files copied to the other 2 CN while they are out of round robin DNS.

To rebuild search index with existing cn data (after an upgrade, if not clearing CN documents/data):

$ sudo nohup /usr/share/dataone-cn-index/scripts/index-tool.sh &

To Copy index data files from one CN to another (to avoid rebuilding on each CN) COPY INDEX FILES: (example shows moving index data files from unm to orc)

$ ssh -A user@cn-stage-unm-1.dataone.org (from workstation to search index source machine, -A forwards credentials)
## stop tc and d1-index-task-generator, d1-index-task-processor (d1-processing if running) deamons on source CN (stops mutations of index data files)
$ cp -r /var/lib/solr/data/d1-cn-index/index/ /home/user/indexCopy (indexCopy is a temp dir, can be removed after rsync)
## start tc and d1-index-task-generator, d1-index-task-processor, (d1-processing) daemons on source CN
$ rsync -av --partial /home/user/indexCopy user@cn-stage-orc-1.dataone.org:/home/user/indexCopy (copy from source to target machine, over 4GB of files to move so may take a bit)
## stop tc and d1-index-task-generator, d1-index-task-processor (d1-processing if running) deamons on target CN (cn-stage-orc-1)
$ rm -rf /var/lib/solr/data/d1-cn-index/index/*  (remove old index data files)
$ cp -r /home/user/indexCopy/* /var/lib/solr/data/d1-cn-index/index/ (copy new data files to solr data dir)
$ sudo chown -R tomcat7.tomcat7 /var/lib/solr/data/d1-cn-index/index/ (assign data files to tomcat7 user/group)
$ start tc, d1-index-task-generator, d1-index-task-processor (d1-processing) daemons on target CN (cn-stage-orc-1).

Previous Steps

The Documentation below is the previous instructions for LVM snapshotting that did not work on Ubuntu 10.04. The instructions are saved for reference for after the upgrade to 12.04.

0.1 Create filesystem backups using LVM snapshots

0.2 Ensure there’s enough space in the LVM Volume Group

The LVM volume group for a set of logical volumes needs enough space to accommodate:

• The growth of each snapshot up to the full size of the original logical volume
• A copy of the the snapshot logical volume (which will replace the current logical volume)
• A minor amount for LVM metadata used to keep track of chunk locations (see lvcreate -s option)

As an example, for a /var data partition of 1TB, we’ll need a volume group with:

• 1TB for the origin /var logical volume
• 1TB for the /var-snapshot logical volume
• 1TB for the copy of /var-snapshot
• 1GB for LVM metadata (not sure about this, probably a conservative estimate)
• Total 3.1TB

The following table shows the current volume group sizes, logical volume sizes, and the needed additions for each CN in the stage and production environments (as of 12Jan2013). In the Required VG Size column, the first value is the minimum needed (2 x VG Size + Snapshot Size + 1G overhead), and the second value is the more liberal needed value ( 3 x VG Size + 1G overhead), as described in the example above. We need to discuss which of these values is most appropriate. The last column, a New VG Size of 4 TB would accommodate either scenario and would let us deal with filesystem growth. Likewise, for simplicity, the three UCSB volume groups could be consolidated into a single volume group like ORC and UNM.

CN	VG Name	VG Size	LV Name	LV Size	Required VG Size	New VG Size
cn-stage-ucsb-1	cn-ucsb-1	46.32 GB	root	38.87 GB	82.74 GB/117.61 GB	4 TB
			snap-root-2012-12-07-1354857208	5.00 GB
	cn-ucsb-1-usr	93.13 GB	usr	93.13 GB	197.26 GB/280.39 GB
			snap-usr-2012-12-07-1354857208	10.00 GB
	cn-ucsb-1-var	931.32 GB	var	931.32 GB	1963.64 GB/2794.96 GB
			snap-var-2012-12-07-1354857208	100.00 GB
cn-stage-orc-1	cn-orc-1-VG	1140.00 GB	cn-orc-1-LV	37.25 GB	2396.00 GB/3421.00 GB	4 TB
			snap-root-2012-12-07-1354857208	5.00 GB
			cn-orc-1-LV2	93.13 GB
			snap-usr-2012-12-07-1354857208	10.00 GB
			cn-orc-1-LV3	931.32 GB
			snap-var-2012-12-07-1354857208	100.00 GB
cn-stage-unm-1	ubuntu	1110.00 GB	root	37.25 GB	2336.00 GB/3331.00 GB	4 TB
			snap-root-2012-12-07-1354857208	5.00 GB
			usr	93.13 GB
			snap-usr-2012-12-07-1354857208	10.00 GB
			var	931.32 GB
			snap-var-2012-12-07-1354857208	100.00 GB
cn-ucsb-1	cn-ucsb-1	46.32 GB	root	38.87 GB	82.74 GB/117.61 GB	4 TB
			snap-root-2013-01-1X-XXXXXXXXXX	5.00 GB
	cn-ucsb-1-usr	93.13 GB	usr	93.13 GB	197.26 GB/280.39 GB
			snap-usr-2013-01-1X-XXXXXXXXXX	10.00 GB
	cn-ucsb-1-var	931.32 GB	var	931.32 GB	1963.64 GB/2794.96 GB
			snap-var-2013-01-1X-XXXXXXXXXX	100.00 GB
cn-orc-1	cn-orc-1-VG	1140.00 GB	cn-orc-1-LV	37.25 GB	2396.00 GB/3421.00 GB	4 TB
			snap-root-2013-01-1X-XXXXXXXXXX	5.00 GB
			cn-orc-1-LV2	93.13 GB
			snap-usr-2013-01-1X-XXXXXXXXXX	10.00 GB
			cn-orc-1-LV3	931.32 GB
			snap-var-2013-01-1X-XXXXXXXXXX	100.00 GB
cn-unm-1	ubuntu	1110.00 GB	root	37.25 GB	2336.00 GB/3331.00 GB	4 TB
			snap-root-2013-01-1X-XXXXXXXXXX	5.00 GB
			usr	93.13 GB
			snap-usr-2013-01-1X-XXXXXXXXXX	10.00 GB
			var	964.16 GB
			snap-var-2013-01-1X-XXXXXXXXXX	100.00 GB

If there isn’t enough space on a given VM for the above:

1. Add new physical (or virtual) disks to the VM (contact Nick Brand, Jamin Ragle, Chris Brumgard)

b) Rescan the SCSI bus to inform the kernel of the new device

# echo "- - -" > /sys/class/scsi_host/host#/scan
(Where # is the host scsi adapter [0,1,2,...]. Rescan all if need be.)

c) For understanding LUN to kernel device name mapping, install lsscsi

# apt-get install lsscsi (This should go into our Ansible config)

# lsscsi
[1:0:0:0]    cd/dvd  NECVMWar VMware IDE CDR10 1.00  /dev/sr0
[2:0:0:0]    disk    VMware   Virtual disk     1.0   /dev/sda
[2:0:1:0]    disk    VMware   Virtual disk     1.0   /dev/sdb
[2:0:2:0]    disk    VMware   Virtual disk     1.0   /dev/sdc

The newly added physical disk will likely be at /dev/sdb or /dev/sdc

d) Create LVM partitions with the new disk (optional, can use whole disk too)

# fdisk /dev/sdb

In fdisk, choose:
n (new partition)
p (primary)
1 (choose partition size by sectors)
t (set the partition type)
8e (Linux LVM)
w (write the partition table)

5. pvcreate /dev/sdb (or /dev/sdb1 if you created a partition)
6. vgextend ubuntu /dev/sdb (Note that the VG on cn-stage-unm-1 is ‘ubuntu’. Others vary)

0.3 Create the snapshot at the rollback instance before any software upgrade

Run the LVM snapshot wrapper script:

# nohup lvm-snapshot.sh > lvm-snapshot.out 2> lvm-snapshot.err < /dev/null &

0.4 In the Stage environment, roll back to the previous stable release

During an upgrade release of the production environment, we first test the installations out in the stage environment. We’ll snapshot the stage VMs, create a 1_X_BRANCH in subversion, install that branch in the stage environment, and test it. If testing succeeds, we create a 1_X_TAG in subversion, and will roll the stage environment back to it’s stable state to emulate the production upgrade. The following procedures describe this rollback.

0.4.1 Backup the /boot filesystem

The /boot filesytem is not an LVM volume, and it will be overwritten when the root logical volume is rolled back. As root, Back it up with dd.

root@cn-stage-unm-1:~# df /boot
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1               233191     35961    184789  17% /boot
# umount /boot
# dd if=/dev/sda1 of=dev_sda1_backup (replace /dev/sda1 with /dev/vda1 for ucsb hosts)
# mount /boot

Once /boot is backed up, rsync the backup to another host to ensure it can be restored

0.4.2 Rename the logical volumes for /, /usr, and /var

By renaming the logical volumes, we can replace them with the snapshot versions taken earlier without booting off of external media. Note that UCSB and ORC volume groups are named differently, so change the rename commands appropriately. List the logical volume names and their logical extent sizes for later use.

# lvdisplay | egrep "LV Name|Current LE"
LV Name                /dev/ubuntu/swap_1
Current LE             11576
LV Name                /dev/ubuntu/root
Current LE             9536
LV Name                /dev/ubuntu/usr
Current LE             23841
LV Name                /dev/ubuntu/var
Current LE             246825
LV Name                /dev/ubuntu/snap-root-2012-12-07-1354857095
Current LE             9536
LV Name                /dev/ubuntu/snap-usr-2012-12-07-1354857095
Current LE             23841
LV Name                /dev/ubuntu/snap-var-2012-12-07-1354857095
Current LE             246825

# lvrename /dev/ubuntu/root /dev/ubuntu/root_old
# lvrename /dev/ubuntu/usr  /dev/ubuntu/usr_old
# lvrename /dev/ubuntu/var  /dev/ubuntu/var_old

0.4.3 Create the replacement logical volumes for the originals

We’ll be copying the snapshot instance to the logical volume names that were originally used, so create the same size volume as what was listed before.

# lvcreate -l 9536   -n root /dev/ubuntu
# lvcreate -l 23841  -n usr  /dev/ubuntu
# lvcreate -l 246825 -n var  /dev/ubuntu

0.4.4 Push the snapshot contents into the new logical volumes

We effectively copy the snapshot volume bytes into a new logical volume. We do this to avoid performance problems that have been documented with LVM snapshot volumes in comparison to non-snapshot volumes. Some tests show an order of magnitude decrease in write performance, so we’re trying to avoid that. Not that block size will affect performance, and a block size of 1M is suggested based on transfer rates at UCSB.

# dd if=/dev/ubuntu/snap-root-2012-12-07-1354857095 of=/dev/ubuntu/root bs=1M
# dd if=/dev/ubuntu/snap-usr-2012-12-07-1354857095  of=/dev/ubuntu/usr  bs=1M
# dd if=/dev/ubuntu/snap-var-2012-12-07-1354857095  of=/dev/ubuntu/var  bs=1M

0.4.5 Restore the /boot filesystem

Copy the bytes of the /boot backup to it’s original partition.

# umount /boot
# dd if=dev_sda1_backup of=/dev/sda1 (dev_vda1_backup and /dev/vda1 for UCSB hosts)
# mount /boot

0.5. Reboot the VM

Now that we’ve replaced the original logical volumes with the snapshot data, bring the VM up with the new volumes as active.

# shutdown -r now

0.6. Remove unneeded logical volumes

To free up space in the volume group for this procedure to be done again, remove both the original logical volume and the snapshot

# lvremove /dev/ubuntu/root_old
# lvremove /dev/ubuntu/usr_old
# lvremove /dev/ubuntu/var_old

# lvremove /dev/ubuntu/snap-root-2012-12-07-1354857095
# lvremove /dev/ubuntu/snap-usr-2012-12-07-1354857095
# lvremove /dev/ubuntu/snap-var-2012-12-07-1354857095