Create the Solr Cloud on the CN clusterΒΆ

A service provided currently by the DataONE cn_common component is a layer for reporting audit issues to an audit log. This audit log is currently implemented with a solr 4 cloud. This allows processing results to be stored and searchable on any/all of the CN cluster members. The solr 4 cloud handles replicating each result to each member of the cloud. This allows clients of the audit log to write a log entry to just one member of the cloud (localhost) and have the audit log message available for reporting or searching on any/all members of the CN cluster.

The DataONE component ‘dataone-cn-audit-index’ is used to install the audit log. This installation depends on a couple of dataone component installs. These are jetty, solr, solr-jetty and zookeeper. The control file for dataone-cn-audit-index defines these dependencies and will install them if not already installed.

Jetty is a servlet container which has the solr webapplication and configuration installed into it. The ‘dataone solr-jetty’ install handles deploying the solr 4 webapp into the jetty container - and also install the necessary jetty context files.

Apache zookeeper is used to manage the solr cloud. Each instance of the solr cloud will connect to the zookeeper server quorum to retrieve the index configuration for the cn-audit index. Zookeeper also manages replication between the solr cloud instances. Zookeeper will also handle recovery operations when a solr instance leaves and re-joins the cloud. Each member of the CN cluster will be run an instance of the zookeeper server. These instances are known as a zookeeper quorum. The quorum operates in majority voting to make decisions - to elect the shard leader for the solr cloud for example. If a majority of zookeeper members are not present, a majority cannot be reached and the solr cloud will not be available - until a majority consensus can be reached. This majority requirement only applies to zookeeper server instances - not solr cloud instances. So if a majority of zookeeper instances are communicating, they will elect a solr cloud shard leader even if only one solr cloud member is active. When other solr instances become available (and in communication with the zookeeper/cloud), zookeeper will manage bringing the new solr instance into ‘sync’ with the cloud - replicating any missing records.

The ‘dataone-cn-audit-index’ component installation scripts will handle configuration of firewall ufw rules that allow the zookeeper servers to communicate. This is done by opening firewall ports and configuring the zoo.cfg configuration file on each cluster member. The installation project contains a file - zoo.cfg which contains tokens that will be substituted with actual values at install time. For example zoo.cfg contains the following lines:

clientPort=D1_ZK_CLIENT_PORT

server.1=D1_CN_IP_1:D1_ZK_SERVER_PORT_1:D1_ZK_SERVER_PORT_2 server.2=D1_CN_IP_2:D1_ZK_SERVER_PORT_1:D1_ZK_SERVER_PORT_2 server.3=D1_CN_IP_3:D1_ZK_SERVER_PORT_1:D1_ZK_SERVER_PORT_2

The values for D1_ZK_CLIENT, D1_ZK_SERVER_PORT_1, and D1_ZK_SERVER_PORT_2 are defined in the installation ‘postinst’ script and should match the ports opened by the ufw rules also present in the installation script.

The ‘dataone-cn-audit-index’ installation will also handle configuration of jetty startup parameters. These parameters are needed to indicate to jetty/solr that a zookeeper quorum is responsible for providing the solr index’s configuration. These parameters are contained in the buildout project file called jetty.start.ini and is deployed to the jetty home directory as start.ini. The parameters are:

-DzkHost=D1_CN_IP_1:D1_ZK_CLIENT_PORT,D1_CN_IP_2:D1_ZK_CLIENT_PORT,D1_CN_IP_3:D1_ZK_CLIENT_PORT

D1_CN_IP_1-3, D1_ZK_CLIENT_PORT values will be substituted with actual IP and port numbers as configured in the installation script. These port values should confirm to the port number opened in the ufw rule.

However the initial bootstrap of index configuration into zookeeper is a manual process at this time. Bootstrap needs to be done for the first solr instance that is being started up only. The jetty start.ini file provided by the ‘d1-cn-audit-index’ component will contain the bootstrap parameters but will be commented out. For exmaple:

# zookeeper boostrap configuration -Dbootstrap_confdir=/var/solr/home/cn-audit/conf -Dcollection.configName=cn-audit -DnumShards=1

These parameters need to be uncommented for the first jetty/solr instance in the cloud to be started. This will push the configuration for the cloud index to zookeeper - for the other members of the cloud to discover and use. Once jetty has been started with these parameters one time, these parameters should be commented back out. Otherwise the next time the instance of jetty was started, the bootstrap process would happen again.

For any other instance of solr that is to join the cloud, nothing manual needs to be done. Just need to start jetty with the provided -DzkHost parameter and the solr instance will join the solr cloud.

This material is based upon work supported by the National Science Foundation under Grant Numbers 083094 and 1430508.

Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.