Troubleshooting CN Cluster Communications

The following steps can be followed to verify that communication between CN instances are working.

Note that ports need to be restored before any of the tests can be performed.

This is done with the following command:

$ sudo /usr/local/bin/togglePortsAndReplication.sh enable

Alternatively, you can check the port status a little crudely with the following.

$ # get the ports that are toggled
$ grep PORT /usr/local/bin/togglePortsAndReplication.sh
PORTS=(5701 5702 5703 389 5432)
$ ufw status

and look for rules for the ports listed.

1. Confirm Certificate Configuration

All DataONE CN components use the same two certificates for inter-CN communications. If you have already confirmed that LDAP synchronization works via resetting a Node attribute, you can assume that the certifcates are installed correctly.

Otherwise:

1.1 Check the Java trust manager

check the java keystore with the following:

$ ps -ef | grep tomcat
tomcat7  12522    1  9 Mar12 ?        02:13:00 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Djava.util.logging.config.file=/var/lib/tomcat …
$ cd /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/security

$ sudo keytool -list -v  -keystore cacerts | grep -A 10 -B 10 DataONE
*******************************************
*******************************************
Alias name: dataoneca
Creation date: Jun 3, 2014
Entry type: trustedCertEntry
Owner: CN=DataONE Test CA, DC=dataone, DC=org
Issuer: CN=DataONE Test CA, DC=dataone, DC=org
Serial number: da3263a2a12d0000
Valid from: Thu Mar 08 03:01:13 UTC 2012 until: Sat Feb 13 03:01:13 UTC 2112
Certificate fingerprints:
MD5:  A4:85:56:5D:F2:B3:C7:2D:13:BA:63:24:AA:E2:90:D5
SHA1: 61:0D:A7:B9:11:AB:BB:0F:6D:B4:47:17:39:C6:53:53:C9:1B:5D:39
SHA256: B6:AD:7F:13:1D:56:EF:D9:5C:E6:27:3E:2E:4C:D3:ED:39:68:0D:59:CC:CE:82:34:93:DD:83:F2:09:4C:83:D7
Signature algorithm name: SHA1withRSA
Version: 3
Extensions:
--
*******************************************
*******************************************
Alias name: debian:dataonetestintca.pem
Creation date: May 30, 2014
Entry type: trustedCertEntry
Owner: CN=DataONE Test Intermediate CA, DC=dataone, DC=org
Issuer: CN=DataONE Test CA, DC=dataone, DC=org
Serial number: da3263a2a12d0049
Valid from: Tue Jul 24 03:24:46 UTC 2012 until: Thu Jun 30 03:24:46 UTC 2112
Certificate fingerprints:
MD5:  3F:52:FC:44:99:DA:7C:7F:9C:9A:90:95:2B:07:9B:4B
SHA1: 97:5B:F3:E8:57:89:9D:B1:3D:FA:64:36:FC:23:C4:4F:46:E8:B5:DC
SHA256: 79:07:78:4B:44:AD:9D:48:16:83:F5:F1:34:29:41:68:3A:EC:E3:0D:0E:AB:C2:3A:C7:9F:B8:6A:8C:8C:94:A9
Signature algorithm name: SHA1withRSA
Version: 3
Extensions:

The Alias name and Creation date values do not matter. Look for:

  • the presence of two entries, the root certificate and the intermediate (see Owner and Issuer attributes).
  • Certificate expiration, the Valid from: and until: dates
  • fingerprint consistency between CNs, it might mean something is amiss.

1.2 Check the Certificate

The CNs maintain both a client certificate and server certificate, in a standard location that all CN subcomponents use for communication. Check that the certficates are in the expected location, and the basic information is correct (Issuer, Validity, and Subject. Use the following example as a guide to locating and inspecting the certifiates.

$ sudo su
$ cd /etc/dataone/client
$ # inspect the server cert
$ openssl x509 -in certs/cn-dev-orc-1.test.dataone.org.pem -text -noout
Certificate:
  Data:
    Version: 3 (0x2)
    Serial Number: 15722738799243755648 (0xda3263a2a12d0080)
  Signature Algorithm: sha1WithRSAEncryption
    Issuer: DC=org, DC=dataone, CN=DataONE Test Intermediate CA
    Validity
        Not Before: Mar 11 18:05:21 2014 GMT
        Not After : Mar 10 18:05:21 2017 GMT
    Subject: DC=org, DC=dataone, CN=cn-dev-orc-1.test.dataone.org
    Subject Public Key Info:
        Public Key Algorithm: rsaEncryption
            Public-Key: (2048 bit)
            Modulus:
                00:aa:8c:96:a6:fa:91:73:c7:6d:e7:43:bf:2a:a4:
                ... etc ...
            Exponent: 65537 (0x10001)
    X509v3 extensions:
        X509v3 Basic Constraints:
            CA:FALSE
        Netscape Comment:
            OpenSSL Generated Certificate
        X509v3 Subject Key Identifier:
            9A:C1:CD:1C:34:7F:30:C3:F4:9E:DC:E0:A9 ... etc ...
        X509v3 Authority Key Identifier:
            keyid:EF:2E:C1:27:6C:2A:8A:09 ... etc ...

        X509v3 CRL Distribution Points:

            Full Name:
              URI:http://releases.dataone.org/crl/DataONETestInt_CRL.pem

            Full Name:
              URI:http://cn-ucsb-1.dataone.org/crl/DataONETestInt_CRL.pem

            Full Name:
              URI:http://cn-unm-1.dataone.org/crl/DataONETest_CRL.pem

            Full Name:
              URI:http://cn-orc-1.dataone.org/crl/DataONETestInt_CRL.pem

  Signature Algorithm: sha1WithRSAEncryption
     23:90:cc:05:a0:e5:b1:2b:11:dc:ee:9a:9b:4d:27:1d:e1:54:
     a9:9e:16:11:9d:64:cf:a6:7d:fd:7d:7d:0f:d0:d9:56:81:33:
     ... etc ...

$ # inspect the client cert
$ sudo openssl x509 -in private/urn_node_cnDevORC1.pem -text -noout
Certificate:
  Data:
    Version: 3 (0x2)
    Serial Number: 15722738799243755649 (0xda3263a2a12d0081)
  Signature Algorithm: sha1WithRSAEncryption
    Issuer: DC=org, DC=dataone, CN=DataONE Test Intermediate CA
    Validity
        Not Before: Mar 11 18:06:14 2014 GMT
        Not After : Mar 10 18:06:14 2017 GMT
      Subject: DC=org, DC=dataone, CN=urn:node:cnDevORC1
      Subject Public Key Info:
        Public Key Algorithm: rsaEncryption
            Public-Key: (2048 bit)
            Modulus:
                00:b6:db:aa:63:33:74:3d:1c:8d:1e:ec:1d:e4:3e:
                71:11:e8:f8:0d:ce:fe:32:87:c3:f0:07:d2:b1:4d:
                ... etc ...
            Exponent: 65537 (0x10001)
      X509v3 extensions:
        X509v3 Basic Constraints:
            CA:FALSE
        Netscape Comment:
            OpenSSL Generated Certificate
        X509v3 Subject Key Identifier:
            BB:63:8E:63:80:2F:15:3E:42:F8:06:2F:F0:DC:9C:45:32:28:32:70
        X509v3 Authority Key Identifier:
            keyid:EF:2E:C1:27:6C:2A:8A:09:AB:6C:C3:45:7F:3B:F9:57:D5:16:A9:B3

        X509v3 CRL Distribution Points:

            Full Name:
              URI:http://releases.dataone.org/crl/DataONETestInt_CRL.pem

            Full Name:
              URI:http://cn-ucsb-1.dataone.org/crl/DataONETestInt_CRL.pem

            Full Name:
              URI:http://cn-unm-1.dataone.org/crl/DataONETest_CRL.pem

            Full Name:
              URI:http://cn-orc-1.dataone.org/crl/DataONETestInt_CRL.pem

  Signature Algorithm: sha1WithRSAEncryption
     46:06:23:fa:97:b6:8e:8e:ee:b0:c4:78:d0:dd:3f:d8:9f:c1:
     7b:38:4c:af:8a:ea:33:43:20:dd:41:b6:3f:63:08:62:4f:12:
     ... etc ...

2. Debug SSL Connection issues

If the certificates seem ok, then you might need to observe the connection negotiating in action. To get a better idea of where SSL handshake issues are failing, start by adding the following to /usr/share/tomcat7/bin/catalina.sh:

$ sudo pico /usr/share/tomcat7/catalina.sh

# add this line:
JAVA_OPTS="$JAVA_OPTS -Djavax.net.debug=ssl:handshake"

and restart tomcat.

This will provide verbose output for all of the steps of the SSL handshake, but not include the listing of all of the certificates registered to the trust-manager, which is likely to be quite verbose: 500 CAs x 20 lines each....

To get that information, omit the :handshake specifier in catalina.sh, using this instead:

JAVA_OPTS="$JAVA_OPTS -Djavax.net.debug=ssl"

This material is based upon work supported by the National Science Foundation under Grant Numbers 083094 and 1430508.

Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.