========================================= Troubleshooting CN Cluster Communications ========================================= The following steps can be followed to verify that communication between CN instances are working. Note that ports need to be restored before any of the tests can be performed. This is done with the following command: :: $ sudo /usr/local/bin/togglePortsAndReplication.sh enable Alternatively, you can check the port status a little crudely with the following. :: $ # get the ports that are toggled $ grep PORT /usr/local/bin/togglePortsAndReplication.sh PORTS=(5701 5702 5703 389 5432) $ ufw status and look for rules for the ports listed. 1. Confirm Certificate Configuration ===================================== All DataONE CN components use the same two certificates for inter-CN communications. If you have already confirmed that LDAP synchronization works via resetting a Node attribute, you can assume that the certifcates are installed correctly. Otherwise: 1.1 Check the Java trust manager -------------------------------- check the java keystore with the following: :: $ ps -ef | grep tomcat tomcat7 12522 1 9 Mar12 ? 02:13:00 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Djava.util.logging.config.file=/var/lib/tomcat … $ cd /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/security $ sudo keytool -list -v -keystore cacerts | grep -A 10 -B 10 DataONE ******************************************* ******************************************* Alias name: dataoneca Creation date: Jun 3, 2014 Entry type: trustedCertEntry Owner: CN=DataONE Test CA, DC=dataone, DC=org Issuer: CN=DataONE Test CA, DC=dataone, DC=org Serial number: da3263a2a12d0000 Valid from: Thu Mar 08 03:01:13 UTC 2012 until: Sat Feb 13 03:01:13 UTC 2112 Certificate fingerprints: MD5: A4:85:56:5D:F2:B3:C7:2D:13:BA:63:24:AA:E2:90:D5 SHA1: 61:0D:A7:B9:11:AB:BB:0F:6D:B4:47:17:39:C6:53:53:C9:1B:5D:39 SHA256: B6:AD:7F:13:1D:56:EF:D9:5C:E6:27:3E:2E:4C:D3:ED:39:68:0D:59:CC:CE:82:34:93:DD:83:F2:09:4C:83:D7 Signature algorithm name: SHA1withRSA Version: 3 Extensions: -- ******************************************* ******************************************* Alias name: debian:dataonetestintca.pem Creation date: May 30, 2014 Entry type: trustedCertEntry Owner: CN=DataONE Test Intermediate CA, DC=dataone, DC=org Issuer: CN=DataONE Test CA, DC=dataone, DC=org Serial number: da3263a2a12d0049 Valid from: Tue Jul 24 03:24:46 UTC 2012 until: Thu Jun 30 03:24:46 UTC 2112 Certificate fingerprints: MD5: 3F:52:FC:44:99:DA:7C:7F:9C:9A:90:95:2B:07:9B:4B SHA1: 97:5B:F3:E8:57:89:9D:B1:3D:FA:64:36:FC:23:C4:4F:46:E8:B5:DC SHA256: 79:07:78:4B:44:AD:9D:48:16:83:F5:F1:34:29:41:68:3A:EC:E3:0D:0E:AB:C2:3A:C7:9F:B8:6A:8C:8C:94:A9 Signature algorithm name: SHA1withRSA Version: 3 Extensions: The Alias name and Creation date values do not matter. Look for: - the presence of two entries, the root certificate and the intermediate (see Owner and Issuer attributes). - Certificate expiration, the Valid from: and until: dates - fingerprint consistency between CNs, it might mean something is amiss. 1.2 Check the Certificate ------------------------- The CNs maintain both a client certificate and server certificate, in a standard location that all CN subcomponents use for communication. Check that the certficates are in the expected location, and the basic information is correct (Issuer, Validity, and Subject. Use the following example as a guide to locating and inspecting the certifiates. :: $ sudo su $ cd /etc/dataone/client $ # inspect the server cert $ openssl x509 -in certs/cn-dev-orc-1.test.dataone.org.pem -text -noout Certificate: Data: Version: 3 (0x2) Serial Number: 15722738799243755648 (0xda3263a2a12d0080) Signature Algorithm: sha1WithRSAEncryption Issuer: DC=org, DC=dataone, CN=DataONE Test Intermediate CA Validity Not Before: Mar 11 18:05:21 2014 GMT Not After : Mar 10 18:05:21 2017 GMT Subject: DC=org, DC=dataone, CN=cn-dev-orc-1.test.dataone.org Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: 00:aa:8c:96:a6:fa:91:73:c7:6d:e7:43:bf:2a:a4: ... etc ... Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Basic Constraints: CA:FALSE Netscape Comment: OpenSSL Generated Certificate X509v3 Subject Key Identifier: 9A:C1:CD:1C:34:7F:30:C3:F4:9E:DC:E0:A9 ... etc ... X509v3 Authority Key Identifier: keyid:EF:2E:C1:27:6C:2A:8A:09 ... etc ... X509v3 CRL Distribution Points: Full Name: URI:http://releases.dataone.org/crl/DataONETestInt_CRL.pem Full Name: URI:http://cn-ucsb-1.dataone.org/crl/DataONETestInt_CRL.pem Full Name: URI:http://cn-unm-1.dataone.org/crl/DataONETest_CRL.pem Full Name: URI:http://cn-orc-1.dataone.org/crl/DataONETestInt_CRL.pem Signature Algorithm: sha1WithRSAEncryption 23:90:cc:05:a0:e5:b1:2b:11:dc:ee:9a:9b:4d:27:1d:e1:54: a9:9e:16:11:9d:64:cf:a6:7d:fd:7d:7d:0f:d0:d9:56:81:33: ... etc ... $ # inspect the client cert $ sudo openssl x509 -in private/urn_node_cnDevORC1.pem -text -noout Certificate: Data: Version: 3 (0x2) Serial Number: 15722738799243755649 (0xda3263a2a12d0081) Signature Algorithm: sha1WithRSAEncryption Issuer: DC=org, DC=dataone, CN=DataONE Test Intermediate CA Validity Not Before: Mar 11 18:06:14 2014 GMT Not After : Mar 10 18:06:14 2017 GMT Subject: DC=org, DC=dataone, CN=urn:node:cnDevORC1 Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: 00:b6:db:aa:63:33:74:3d:1c:8d:1e:ec:1d:e4:3e: 71:11:e8:f8:0d:ce:fe:32:87:c3:f0:07:d2:b1:4d: ... etc ... Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Basic Constraints: CA:FALSE Netscape Comment: OpenSSL Generated Certificate X509v3 Subject Key Identifier: BB:63:8E:63:80:2F:15:3E:42:F8:06:2F:F0:DC:9C:45:32:28:32:70 X509v3 Authority Key Identifier: keyid:EF:2E:C1:27:6C:2A:8A:09:AB:6C:C3:45:7F:3B:F9:57:D5:16:A9:B3 X509v3 CRL Distribution Points: Full Name: URI:http://releases.dataone.org/crl/DataONETestInt_CRL.pem Full Name: URI:http://cn-ucsb-1.dataone.org/crl/DataONETestInt_CRL.pem Full Name: URI:http://cn-unm-1.dataone.org/crl/DataONETest_CRL.pem Full Name: URI:http://cn-orc-1.dataone.org/crl/DataONETestInt_CRL.pem Signature Algorithm: sha1WithRSAEncryption 46:06:23:fa:97:b6:8e:8e:ee:b0:c4:78:d0:dd:3f:d8:9f:c1: 7b:38:4c:af:8a:ea:33:43:20:dd:41:b6:3f:63:08:62:4f:12: ... etc ... 2. Debug SSL Connection issues ================================ If the certificates seem ok, then you might need to observe the connection negotiating in action. To get a better idea of where SSL handshake issues are failing, start by adding the following to /usr/share/tomcat7/bin/catalina.sh: :: $ sudo pico /usr/share/tomcat7/catalina.sh # add this line: JAVA_OPTS="$JAVA_OPTS -Djavax.net.debug=ssl:handshake" and restart tomcat. This will provide verbose output for all of the steps of the SSL handshake, but not include the listing of all of the certificates registered to the trust-manager, which is likely to be quite verbose: 500 CAs x 20 lines each.... To get that information, omit the :handshake specifier in catalina.sh, using this instead: :: JAVA_OPTS="$JAVA_OPTS -Djavax.net.debug=ssl" - A good resource for analyzing the output is: http://www.smartjava.org/content/how-analyze-java-ssl-errors - SSL debug output will be part of the log file for the affected applications. - Be sure to turn-off SSL debugging when you are done, since it is quite verbose in the log files.