Create a Coordinating Node
==========================
This document describes how to create a new Coordinating Node and configure it
to replicate with other existing Coordinating Nodes.
Quick steps:
1. Create the Coordinating Node virtual machine
2. Install the Coordinating Node software stack
3. Configure the software stack
Create the Virtual Machine
--------------------------
Coordinating Nodes are currently being built on Ubuntu 10.04 LTS.
Details for creating KVM or OpenVZ virtual machines are detailed in the
document :doc:`vm_creation`. Coordinating Nodes should be provisioned with
significant resources.
.. list-table::
:header-rows: 1
* - Type
- RAM
- Cores
- SWAP
- Disk space
* - Development CN
- 4096 MB
- 2
- 1024
- 100000 MB
* - Production CN
- 32768 MB
- 8
- 2048
- 300000 MB (or more)
After creating the new virtual machine, enable the *partner* repository by
uncommenting or adding the following lines in ``/etc/apt/sources.list.d/dataone.list``::
deb http://archive.canonical.com/ubuntu precise partner
deb-src http://archive.canonical.com/ubuntu precise partner
The following additional packages (beyond the base install described in
:doc:`create_vm`) should then be installed::
You may install a handy software package that allows ease of creating source listing entries
for apt-get, though it only works for additions to ``/etc/apt/sources.list``, not contents of ``/etc/apt/sources.list.d/``::
sudo apt-get install python-software-properties
The CN software stack is maintained in an *apt* repository. To install the
stack, add the following entries in ``/etc/apt/sources.list.d/dataone.list``::
deb [arch=amd64] http://jenkins-1.dataone.org/ubuntu-unstable precise universe
or
deb [arch=amd64] http://jenkins-1.dataone.org/ubuntu-stable precise universe
and
sudo add-apt-repository "deb [arch=amd64] http://jenkins-1.dataone.org/ubuntu-extra precise universe"
Ensure the system is up to date before continuing::
sudo apt-get update
sudo apt-get upgrade
Lastly, The following packages will need to be installed in this order::
sudo apt-get --no-install-recommends install openjdk-7-jre-headless
sudo update-java-alternatives -s java-7-openjdk-amd64
sudo apt-get --no-install-recommends install libjaxp1.3-java
sudo apt-get --no-install-recommends install libxerces2-java
sudo apt-get --no-install-recommends install ant
sudo apt-get install ssl-cert
The *--no-install-recommends* switch prevents installation of the Ubuntu
preferred JVM (gcj) and so ensures that openjdk-7-jre-headless remains the only JVM
available on the system. Unintended installation of gcj appears to have been
the cause of some unanticipated behavior of the CN stack.
Install the Coordinating Node Software
--------------------------------------
Before proceeding with installation of the CN software stack, it is necessary
to install a certificate that will be used for enabling HTTPS communications.
If the CN has a host name in the *dataone.org* domain, then the DataONE wild
card certificate may be used. This may be retrieved from any existing
Coordinating Node or alternatively, contact an administrator (see
:doc:`administration-contacts`). The wild card certificate key should be
copied into /etc/ssl/private/dataone_org.key. If the CN has a host name in
the *.test.dataone.org* domain, then the DataONE Test wild
card certificate may be used. The test wild card certificate key should be
copied into /etc/ssl/private/_.test.dataone_org.key.
Alternatively, a self-signed CN certificate can be used for an individual developer's virtual
development environment for testing and development. The procedure for
creating and installing a self-signed certificate goes something like::
cd
pwd
/home/waltz
mkdir ssl
cd ssl
openssl genrsa -des3 -out server.key 1024
Generating RSA private key, 1024 bit long modulus
...........................................++++++
.............++++++
e is 65537 (0x10001)
Enter pass phrase for server.key: <type something here and remember it>
Verifying - Enter pass phrase for server.key: <type something here and remember it>
chmod 600 server.key
openssl req -new -key server.key -out server.csr
Enter pass phrase for server.key: <type something here that you typed above>
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:Tennessee
Locality Name (eg, city) []:Knoxville
Organization Name (eg, company) [Internet Widgits Pty Ltd]:DataONE.org
Organizational Unit Name (eg, section) []:
Common Name (eg, YOUR name) []:cn-dev.dataone.org
Email Address []:rwaltz@cn-dev
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []: <pressed enter>
An optional company name []: <pressed enter>
XXX NOTE that the common name is spurious XXX
cp server.key server.key.orig
openssl rsa -in server.key.orig -out server.key
Enter pass phrase for server.key.orig: <type something here that you typed above>
writing RSA key
openssl x509 -req -days 3650 -in server.csr -signkey server.key -out server.crt
Signature ok
subject=/C=US/ST=Tennessee/L=Knoxville/O=DataONE.org/CN=cn-dev.utk.edu/emailAddress=rwaltz@cn-dev.utk.edu
Getting Private key
sudo cp server.key /etc/ssl/private/dataone_org.key
sudo cp server.crt /etc/ssl/certs/dataone.org.crt
Next, install the CN software using the usual steps::
sudo apt-get update
sudo apt-get install dataone-cn-os-core dataone-cn-metacat \
dataone-cn-portal dataone-cn-rest-service dataone-cn-processdaemon \
dataone-cn-index dataone-mercury dataone-cn-solr dataone-cn-version-tool
A number of additional packages will be installed as dependencies of the CN
software stack. The current (2015-07-21) list of additional packages is
provided below::
apache2 apache2-mpm-worker apache2-utils apache2.2-bin apache2.2-common authbind
check-mk-agent-logwatch cryptsetup-bin dataone-cn-index dataone-cn-metacat dataone-cn-os-core
dataone-cn-portal dataone-cn-processdaemon dataone-cn-rest-service dataone-cn-solr
dataone-cn-version-tool dataone-mercury dbus-x11 dconf-gsettings-backend dconf-service
debconf-utils fontconfig gawk gconf-service gconf-service-backend gconf2 gconf2-common gvfs
gvfs-common gvfs-daemons gvfs-libs hicolor-icon-theme jsvc ldap-utils libantlr-java
libapache-pom-java libapache2-mod-jk libapr1 libaprutil1 libaprutil1-dbd-sqlite3 libaprutil1-ldap
libasound2 libatasmart4 libatk-wrapper-java libatk-wrapper-java-jni libatk1.0-0 libatk1.0-data
libavahi-glib1 libbackport-util-concurrent-java libbonobo2-0 libbonobo2-common libcairo-gobject2
libcairo2 libcanberra0 libcommons-beanutils-java libcommons-codec-java
libcommons-collections-java libcommons-collections3-java libcommons-csv-java
libcommons-daemon-java libcommons-dbcp-java libcommons-digester-java libcommons-fileupload-java
libcommons-httpclient-java libcommons-io-java libcommons-lang-java libcommons-logging-java
libcommons-parent-java libcommons-pool-java libcommons-validator-java libconvert-asn1-perl
libcryptsetup4 libdatrie1 libdconf0 libdom4j-java libecj-java libencode-locale-perl
libexcalibur-logkit-java libfile-listing-perl libfont-afm-perl libfontenc1 libgconf-2-4
libgconf2-4 libgdk-pixbuf2.0-0 libgdk-pixbuf2.0-common libgdu0 libgif4 libgl1-mesa-dri
libgl1-mesa-glx libglapi-mesa libgnome-keyring-common libgnome-keyring0 libgnome2-0 libgnome2-bin
libgnome2-common libgnomevfs2-0 libgnomevfs2-common libgtk-3-0 libgtk-3-bin libgtk-3-common
libgtk2.0-0 libgtk2.0-bin libgtk2.0-common libgudev-1.0-0 libhtml-form-perl libhtml-format-perl
libhtml-parser-perl libhtml-tagset-perl libhtml-tree-perl libhttp-cookies-perl
libhttp-daemon-perl libhttp-date-perl libhttp-message-perl libhttp-negotiate-perl libice-dev
libidl-common libidl0 libio-socket-inet6-perl libio-socket-ssl-perl libjasper1 libjaxen-java
libjaxme-java libjdom1-java libjetty-java libllvm3.0 liblog4j1.2-java libltdl7 liblvm2app2.2
liblwp-mediatypes-perl liblwp-protocol-https-perl libmailtools-perl libnet-http-perl
libnet-ldap-perl libnet-ssleay-perl libnoggit-java libodbc1 liborbit2 liboro-java libpango1.0-0
libperl5.14 libpixman-1-0 libpolkit-agent-1-0 libpolkit-backend-1-0 libportlet-api-2.0-spec-java
libpq5 libpthread-stubs0 libpthread-stubs0-dev libservlet2.5-java libservlet3.0-java
libsgutils2-2 libsigsegv2 libslf4j-java libslp1 libsm-dev libsocket6-perl libtdb1 libthai-data
libthai0 libtiff4 libtomcat7-java liburi-perl libvelocity-tools-java libvorbisfile3
libwerken.xpath-java libwww-perl libwww-robotrules-perl libx11-dev libx11-doc libx11-xcb1
libxau-dev libxcb-glx0 libxcb-render0 libxcb-shape0 libxcb-shm0 libxcb1-dev libxcomposite1
libxdamage1 libxdmcp-dev libxft2 libxinerama1 libxom-java libxpp2-java libxpp3-java libxslt1.1
libxt-dev libxtst6 libxv1 libxxf86dga1 mtools openjdk-7-jdk openjdk-7-jre policykit-1
policykit-1-gnome postgresql postgresql-9.1 postgresql-client-9.1 postgresql-client-common
postgresql-common shared-mime-info slapd solr-common7 solr-tomcat7 sound-theme-freedesktop
tomcat7 tomcat7-admin tomcat7-common ttf-dejavu-extra udisks velocity x11-utils x11proto-core-dev
x11proto-input-dev x11proto-kb-dev xmlstarlet xorg-sgml-doctools xtrans-dev
Configuring a Coordinating Node
-------------------------------
Temporary fix
chown -R openldap.openldap /var/lib/ldap
Creating a Metacat administrative user in LDAP (production only)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
After installing the debian packages, but prior to configuring Metacat, the
Metacat administrative DN must be added to LDAP for the production environment.
The uid=dataone_cn_metacat,o=DATAONE,dc=ecoinformatics,dc=org DN is used in the
testing environments. To add the new DN:
1) Use slappasswd to create an SHA encrypted version of the password:
$ slappasswd -h {SHA} -s <enter-password-here>
2) Create an ldif file named dataone_cn_metacat.ldif with the password
appended to the end, similar to:
dn: cn=dataone_cn_metacat,dc=dataone,dc=org
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
cn: dataone_cn_metacat
givenName: DataONE
mail: jones@nceas.ucsb.edu
sn: Metacat
userPassword: {SHA}vx7+jWYTWQAjzRvh0owepFFGdi8=
but replace the value of the userPassword with the output of the command above.
3) Add the entry to one of the CNs (say, cn-ucsb-1.dataone.org) with:
$ ldapadd -x -D cn=admin,dc=dataone,dc=org -W -f dataone_cn_metacat.ldif
Configuring Metacat authentication (production only)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
For the production environment, the Metacat administrative user created above
needs to be configured in the Metacat
With apache and tomcat running, go to:
https://<hostname>.dataone.org/knb/admin
The Authentication configuration page should come up.
change the follwing fields with the list values:
Authentication Class
**** Add CN contactSubject into dataone (robert's DN or other) ****
Configuring Solr Cloud
~~~~~~~~~~~~~~~~~~~~~~~~~~
Please see Solr4Cloud_ for details on solr cloud configuration.
Change SSH server to Certificate only
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
All servers will have their remote ssh login by password disabled. The only way
to login with a password is via console.
You should have a copy of your public key available for installation in your .ssh directory.
The following is copied and edited from _Generate a ssh key and disable password authentication on Ubuntu Server 12.04
.. _Generate a ssh key and disable password authentication on Ubuntu Server 12.04: http://lani78.com/2012/07/21/generate-a-ssh-key-and-disable-password-authentication-on-ubuntu-server-12-0/
*Step by Step guide*
Obviously, substitute [your username here] with your username, rsmith for instance.
1) Generate the ssh key pair on your client computer:
> ssh-keygen \
> -t rsa \
> -b 4096 \
> -C "[your username here] d1 key pair" \
> -f ~/.ssh/[your username here]_dataone_rsa
When prompted, create a strong (five+ random word) passphrase. Back this key file up securely in case your drive dies, etc.
2) Transfer your public key onto *each* (sigh) dataone VM you want to access
a) Test your setup from your laptop/workstation
> ssh cn-dev-ucsb-1.test.dataone.org date
b) Copy the public key to the server
> scp ~/.ssh/[your username here]_dataone_rsa.pub [your username here]@cn-dev-ucsb-1.test.dataone.org:[your username here]_dataone_rsa.pub
3) Connect to the server:
> ssh [your username here]@cn-dev-ucsb-1.test.dataone.org
4) Append the public key to authorized_keys and remove the uploaded copy:
> cat [your username here]_dataone_rsa.pub >> ~/.ssh/authorized_keys
> rm [your username here]_dataone_rsa.pub
> chmod 600 .ssh/authorized_keys
> chmod 700 .ssh
5) Edit the ssh server configuration to make sure that public key authentication is enabled (it should be enabled by default):
> sudo nano /etc/ssh/sshd_config
a) These entries must be set to yes:
> RSAAuthentication yes
> PubkeyAuthentication yes
6) Reload the configuration:
> sudo service ssh reload
7) Disconnect from the server:
> exit
8) Try connecting without the need to give the password to the ssh-client:
> ssh [your username here]@cn-dev-ucsb-1.test.dataone.org
You might need to give a password now to access your private key file, but you should not need to give the password to the ssh program.
9) Disable password authentication:
> sudo nano /etc/ssh/sshd_config
a) The following settings should be set to no:
ChallengeResponseAuthentication no
PasswordAuthentication no
UsePAM yes
b) Reload the configuration:
> sudo service ssh reload
10) Test that password authentication really is disabled:
a) Disconnect from the server:
> exit
b) Try to reconnect to the server with key file authentication disabled:
> ssh [your username here]@cn-dev-ucsb-1.test.dataone.org -o PubkeyAuthentication=no
This should produce a permission denied message: “Permission denied (publickey).â€
Updating a Coordinating Node
----------------------------
It will be necessary to update the CN software stack on a regular basis while
the CN software stack is under active development. For the most part, this
process should be as simple as the normal installed software update process
of::
sudo apt-get update
sudo apt-get upgrade
The version information of the cn-buildout packages is automatically updated
as they are built by Hudson_ and so any package updates should be picked up by
*apt*.
When new search fields are added or existing fields modified, it may be necessary
to migrate the data in the current search index into a new index with the updated
schema. This will be indicated by the exit message of the dataone-cn-index
component. Until the migration has completed the previous search index should
remain available for users. The migration script is located at:
/usr/share/dataone-cn-index/scripts/migrate-search-index.sh
and requires sudo permission to run.
It may be necessary on occasion to perform a more extensive update
that can not be effectively handled through the automated process.
Notification of such situations should be made through the
developers@dataone.org mailing list.
.. _Hudson: http://dev-testing.dataone.org:8080/hudson
.. _Solr4Cloud: http://mule1.dataone.org/OperationDocs/solr4_cloud_config.html