Natural History of System Metadata ================================== :Revisions: ======== ================================================================== Date Comment ======== ================================================================== 20100303 (D.V.) Initial draft 20100304 (D.V.) Revised draft and added scenario where system metadata is stored only on the Coordinating Nodes Corrected DataSysMetadataModified changes in scenario B. ======== ================================================================== This document describes the generation of system metadata associated with a data object and its associated science metadata document as they are added to a DataONE Member Node and subsequently propagated through the DataONE infrastructure. .. contents:: :depth: 1 :local: Process Overview ---------------- There are two scenarios examined here: A) the system metadata is stored on all nodes (MN and CN) and B) system metadata is stored only on the CNs. The general sequence of operations is the same for both scenarios with the exclusion of a couple of steps from scenario B. 1. Scientist using an ITK tool generates some science data and some science metadata describing the science data. 2. The ITK populates the required fields of the system metadata documents. 3. Not necessarily in this order: a) The ITK uploads the science data b) The ITK uploads the science metadata On receipt, the MN updates the system metadata fields for which it is responsible. 4. The CN detects new content is available on the MN using the listObjects() method. 5. Copies of the science metadata and the two (incomplete) system metadata documents are retrieved by the CN. :A: The CN updates the system metadata replication information, and indicates to the MN that it should update its copy of the system metadata objects. :B: The CN updates the system metadata replication information. 6. The CN issues a replication order to an MN. :A: The recipient MN retrieves copies of the system metadata and associated objects (science data and metadata) :B: The recipient MN retrieves copies of the various objects indicated by the CN 7. The recipient MN requests content from the origin MN, which in turn verifies the request with the CN. :A: The origin MN and the CN update copies of system metadata :B: The CN updates system metadata 8. After verifying the successful transfer of content, the recipient MN indicates completion to the CN. :A: The CN updates the system metadata for the objects and ensures that the MN copies of the system metadata are accurate. :B: The CN updates the system metadata for the objects. The change in system metadata state is shown in the following process detail sections. In each case, the identifiers used for the various elements are indicated in table 1. .. table:: Table 1. Identifiers used in the example sequence. =============== ================================================ Identifier Description =============== ================================================ sciD.1 Science Data sciM.1 Science metadata describing sciD.1 sysD.1 System metadata for object sciD.1 sysM.1 System metadata for object sciM.1 cn1.dataone.org A Coordinating Node mn1.dataone.org Member Node that is initial recipient of data mn2.dataone.org Member Node that receives a copy of the objects =============== ================================================ The following fictitious script provides an example of how uploading content to a Member Node using a DataONE client library might occur. .. code-block:: python import sys import logging from pyd1.client import * from pyd1.exceptions import * d1client = pyd1.client(identity="/home/vieglais/.d1/credentials", defaults="/home/vieglais/.d1/defaults.ini") d1client.login() # The science data and metadata as files on disk data_file = "/home/vieglais/data/mydata.csv" meta_file = "/home/vieglais/data/mymeta.xml" # Reserve a couple of identifiers. Note that this reservation will # be system-wide. try: data_id = d1client.reserve(u"sysD.1") meta_id = d1client.reserve(u"sysM.1") except IdentifierInUseException, e: logging.fatal(repr(e)) sys.exit() # Generate stub system metadata documents. Default values # for access rules etc are loaded from defaults.ini sysm_data = d1client.generateSysMeta(id=data_id, format="text/csv", describedBy=meta_id) sysm_meta = d1client.generateSysMeta(id=meta_id, format="eml://ecoinformatics.org/eml-2.1.0", describedBy=data_id) # Open file handles for data and metadata fdata = file(data_file, "r") fmetadata = file(meta_file, "r") # Send the data to the Member Node target_node = 'mn1.dataone.org' data_sysid = d1client.create(data_id, fdata, sys_data, target=target_node) meta_sysid = d1client.create(meta_id, fmeta, sys_meta, target=target_node) fdata.close() fmeta.close() Process Detail - Scenario A --------------------------- Step A1. ~~~~~~~~ The scientist generates a data set and associated science metadata. Step A2. ~~~~~~~~ Using tools in the ITK, the scientist generates two initial system metadata documents, one describing the science metadata (``sysm_M.1``), one describing the science data (``sysm_D.1``). =========================== ========================================== ======================== Field sysD.1 sysM.1 =========================== ========================================== ======================== identifier sciD.1 sciM.1 objectFormat text/csv eml://ecoinformatics.org/eml-2.1.0 size 1450238 8132 submitter uid=jones,o=NCEAS,dc=ecoinformatics,dc=org uid=jones,o=NCEAS,dc=ecoinformatics,dc=org rightsHolder uid=jones,o=NCEAS,dc=ecoinformatics,dc=org uid=jones,o=NCEAS,dc=ecoinformatics,dc=org obsoletes \ \ obsoletedBy \ \ derivedFrom \ \ describes \ sciD.1 describedBy sciM.1 \ checksum 2e01e17467891f7c933dbaa00e1459d23db3fe4f 9967467891f7c933dbaa00e1459d23db3f342 checksumAlgorithm SHA-1 SHA-1 accessRule[1] \ \ .rule Allow Allow .service Read Read .principal \* \* accessRule[2] \ \ .rule Allow Allow .service Write Write .principal o=NCEAS,dc=ecoinformatics,dc=org o=NCEAS,dc=ecoinformatics,dc=org replicationPolicy \ \ .preferredMemberNode mn2.dataone.org mn2.dataone.org .blockedMemberNode \ \ .replicationAllowed true true .numberReplicas 1 1 dateUploaded \ \ dateSysMetadataModified \ \ originMemberNode \ \ authoritativeMemberNode \ \ replica[1] \ \ .replicaMemberNode \ \ .replicationStatus \ \ .replicaVerified \ \ =========================== ========================================== ======================== Step A3. ~~~~~~~~ The client tool transmits the data, science metadata and two stub system metadata documents to a Member Node. The Member Node updates values of the system metadata for which it is responsible. =========================== ========================================== ========================================== Field sysD.1 sysM.1 =========================== ========================================== ========================================== identifier sciD.1 sciM.1 objectFormat text/csv eml://ecoinformatics.org/eml-2.1.0 size 1450238 8132 submitter uid=jones,o=NCEAS,dc=ecoinformatics,dc=org uid=jones,o=NCEAS,dc=ecoinformatics,dc=org rightsHolder uid=jones,o=NCEAS,dc=ecoinformatics,dc=org uid=jones,o=NCEAS,dc=ecoinformatics,dc=org obsoletes \ \ obsoletedBy \ \ derivedFrom \ \ describes \ sciD.1 describedBy sciM.1 \ checksum 2e01e17467891f7c933dbaa00e1459d23db3fe4f 9967467891f7c933dbaa00e1459d23db3f342 checksumAlgorithm SHA-1 SHA-1 accessRule[1] \ \ .rule Allow Allow .service Read Read .principal \* \* accessRule[2] \ \ .rule Allow Allow .service Write Write .principal o=NCEAS,dc=ecoinformatics,dc=org o=NCEAS,dc=ecoinformatics,dc=org replicationPolicy \ \ .preferredMemberNode mn2.dataone.org mn2.dataone.org .blockedMemberNode \ \ .replicationAllowed true true .numberReplicas 1 1 dateUploaded **2010-03-04T18:13:51.0Z** **2010-03-04T18:14:45.0Z** dateSysMetadataModified **2010-03-04T18:13:51.0Z** **2010-03-04T18:14:45.0Z** originMemberNode **mn1.dataone.org** **mn1.dataone.org** authoritativeMemberNode **mn1.dataone.org** **mn1.dataone.org** replica[1] \ \ .replicaMemberNode **mn1.dataone.org** **mn1.dataone.org** .replicationStatus **Queued** **Queued** .replicaVerified \ \ =========================== ========================================== ========================================== Since there are many copies of system metadata from this point onwards, only the replication properties of the ``sysD.1`` document are considered, summarized in a table as follows. MN1, CN, MN2 = copy of system metadata on ``mn1.dataone.org``, ``cn1.dataone.org``, ``mn2.dataone.org`` respectively. ========================== ====================== ====================== ====================== Property MN1 CN MN2 ========================== ====================== ====================== ====================== R[1].replicaMemberNode mn1.dataone.org R[1].replicationStatus Queued R[1].replicaVerified ========================== ====================== ====================== ====================== Step A4. ~~~~~~~~ A CN detects new content available on ``mn1.dataone.org`` through the listObjects() method. There is no change of state in system metadata resulting from this operation. Step A5. ~~~~~~~~ Copies of ``sciM.1``, ``sysM.1``, and ``sysD.1`` are retrieved to the CN, and the system metadata objects are updated. There are three steps to this process: 1. The CN requests a copy of the objects. ========================== ====================== ====================== ====================== Property MN1 CN MN2 ========================== ====================== ====================== ====================== R[1].replicaMemberNode mn1.dataone.org R[1].replicationStatus **Requested** R[1].replicaVerified ========================== ====================== ====================== ====================== 2. The CN receives a copy of the metadata. ========================== ====================== ====================== ====================== Property MN1 CN MN2 ========================== ====================== ====================== ====================== R[1].replicaMemberNode mn1.dataone.org **mn1.dataone.org** R[1].replicationStatus Requested **Requested** R[1].replicaVerified ========================== ====================== ====================== ====================== 3. The CN verifies the checksums and indicates to the MN that the copy is complete, providing the time stamp that it used for the verification (ensuring identical copies of the system metadata). ========================== ========================== ========================== ========================== Property MN1 CN MN2 ========================== ========================== ========================== ========================== R[1].replicaMemberNode mn1.dataone.org mn1.dataone.org R[1].replicationStatus **Completed** **Completed** R[1].replicaVerified **2010-03-04T19:13:51.0Z** **2010-03-04T19:13:51.0Z** ========================== ========================== ========================== ========================== Step A6. ~~~~~~~~ The CN initiates replication of content from ``mn1.dataone.org`` to ``mn2.dataone.org``. 1. The CN sends a replication request to MN2, indicating that it should retrieve the object from MN1. ========================== ====================== ====================== ====================== Property MN1 CN MN2 ========================== ====================== ====================== ====================== R[1].replicaMemberNode mn1.dataone.org mn1.dataone.org R[1].replicationStatus Completed Completed R[1].replicaVerified 2010-03-04T19:13:51.0Z 2010-03-04T19:13:51.0Z R[2].replicaMemberNode **mn2.dataone.org** R[2].replicationStatus **Queued** R[2].replicaVerified ========================== ====================== ====================== ====================== 2. MN2 requests the content from MN1. ========================== ====================== ====================== ========================== Property MN1 CN MN2 ========================== ====================== ====================== ========================== R[1].replicaMemberNode mn1.dataone.org mn1.dataone.org **mn1.dataone.org** R[1].replicationStatus Completed Completed **Completed** R[1].replicaVerified 2010-03-04T19:13:51.0Z 2010-03-04T19:13:51.0Z **2010-03-04T19:13:51.0Z** R[2].replicaMemberNode **mn2.dataone.org** mn2.dataone.org **mn2.dataone.org** R[2].replicationStatus **Requested** Queued **Requested** R[2].replicaVerified ========================== ====================== ====================== ========================== 3. MN1 checks with CN that the object was requested ========================== ====================== ====================== ====================== Property MN1 CN MN2 ========================== ====================== ====================== ====================== R[1].replicaMemberNode mn1.dataone.org mn1.dataone.org mn1.dataone.org R[1].replicationStatus Completed Completed Completed R[1].replicaVerified 2010-03-04T19:13:51.0Z 2010-03-04T19:13:51.0Z 2010-03-04T19:13:51.0Z R[2].replicaMemberNode mn2.dataone.org mn2.dataone.org mn2.dataone.org R[2].replicationStatus Requested **Requested** Requested R[2].replicaVerified ========================== ====================== ====================== ====================== 4. MN2 verifies that the checksum is correct, then indicates to MN1 that the copy was correctly transferred, forwarding the timestamp for when the object was verified to MN1. ========================== ========================== ========================== ========================== Property MN1 CN MN2 ========================== ========================== ========================== ========================== R[1].replicaMemberNode mn1.dataone.org mn1.dataone.org mn1.dataone.org R[1].replicationStatus Completed Completed Completed R[1].replicaVerified 2010-03-04T19:13:51.0Z 2010-03-04T19:13:51.0Z 2010-03-04T19:13:51.0Z R[2].replicaMemberNode mn2.dataone.org mn2.dataone.org mn2.dataone.org R[2].replicationStatus **Completed** Requested **Completed** R[2].replicaVerified **2010-03-04T20:00:00.0Z** **2010-03-04T20:00:00.0Z** ========================== ========================== ========================== ========================== Step A8. ~~~~~~~~ MN2 informs the Coordinating Node that it has retrieved a valid copy of the object from MN1, forwarding the time stamp for when the verification was made. ========================== ====================== ========================== ====================== Property MN1 CN MN2 ========================== ====================== ========================== ====================== R[1].replicaMemberNode mn1.dataone.org mn1.dataone.org mn1.dataone.org R[1].replicationStatus Completed Completed Completed R[1].replicaVerified 2010-03-04T19:13:51.0Z 2010-03-04T19:13:51.0Z 2010-03-04T19:13:51.0Z R[2].replicaMemberNode mn2.dataone.org mn2.dataone.org mn2.dataone.org R[2].replicationStatus Completed **Completed** Completed R[2].replicaVerified 2010-03-04T20:00:00.0Z **2010-03-04T20:00:00.0Z** 2010-03-04T20:00:00.0Z ========================== ====================== ========================== ====================== Process Detail - Scenario B. ---------------------------- This scenario uses the notion that system metadata is stored on the CNs only. Step B1. ~~~~~~~~ The scientist generates a data set and associated science metadata. Step B2. ~~~~~~~~ Using tools in the ITK, the scientist generates two initial system metadata documents, one describing the science metadata (``sysm_M.1``), one describing the science data (``sysm_D.1``). =========================== ========================================== ======================== Field sysD.1 sysM.1 =========================== ========================================== ======================== identifier sciD.1 sciM.1 objectFormat text/csv eml://ecoinformatics.org/eml-2.1.0 size 1450238 8132 submitter uid=jones,o=NCEAS,dc=ecoinformatics,dc=org uid=jones,o=NCEAS,dc=ecoinformatics,dc=org rightsHolder uid=jones,o=NCEAS,dc=ecoinformatics,dc=org uid=jones,o=NCEAS,dc=ecoinformatics,dc=org obsoletes obsoletedBy derivedFrom describes sciD.1 describedBy sciM.1 checksum 2e01e17467891f7c933dbaa00e1459d23db3fe4f 9967467891f7c933dbaa00e1459d23db3f342 checksumAlgorithm SHA-1 SHA-1 accessRule[1] .rule Allow Allow .service Read Read .principal \* \* accessRule[2] .rule Allow Allow .service Write Write .principal o=NCEAS,dc=ecoinformatics,dc=org o=NCEAS,dc=ecoinformatics,dc=org replicationPolicy .preferredMemberNode mn2.dataone.org mn2.dataone.org .blockedMemberNode .replicationAllowed true true .numberReplicas 1 1 dateUploaded dateSysMetadataModified originMemberNode authoritativeMemberNode replica[1] .replicaMemberNode .replicationStatus .replicaVerified =========================== ========================================== ======================== Step B3. ~~~~~~~~ The client tool transmits the data, science metadata and two stub system metadata documents to a Member Node (``mn1.dataone.org``). On receipt, the Member Node sets the values of several fields as indicted in the table below. =========================== ========================================== ========================================== Field sysD.1 sysM.1 =========================== ========================================== ========================================== identifier sciD.1 sciM.1 objectFormat text/csv eml://ecoinformatics.org/eml-2.1.0 size 1450238 8132 submitter uid=jones,o=NCEAS,dc=ecoinformatics,dc=org uid=jones,o=NCEAS,dc=ecoinformatics,dc=org rightsHolder uid=jones,o=NCEAS,dc=ecoinformatics,dc=org uid=jones,o=NCEAS,dc=ecoinformatics,dc=org obsoletes obsoletedBy derivedFrom describes sciD.1 describedBy sciM.1 checksum 2e01e17467891f7c933dbaa00e1459d23db3fe4f 9967467891f7c933dbaa00e1459d23db3f342 checksumAlgorithm SHA-1 SHA-1 accessRule[1] .ruleType Allow Allow .service Read Read .principal \* \* accessRule[2] .ruleType Allow Allow .service Write Write .principal o=NCEAS,dc=ecoinformatics,dc=org o=NCEAS,dc=ecoinformatics,dc=org replicationPolicy .preferredMemberNode mn2.dataone.org mn2.dataone.org .blockedMemberNode .replicationAllowed true true .numberReplicas 1 1 dateUploaded **2010-03-04T18:13:51.0Z** **2010-03-04T18:14:45.0Z** dateSysMetadataModified **2010-03-04T18:13:51.0Z** **2010-03-04T18:14:45.0Z** originMemberNode **mn1.dataone.org** **mn1.dataone.org** authoritativeMemberNode **mn1.dataone.org** **mn1.dataone.org** replica[1] .replicaMemberNode **mn1.dataone.org** **mn1.dataone.org** .replicationStatus **Queued** **Queued** .replicaVerified =========================== ========================================== ========================================== Step B4. ~~~~~~~~ A CN detects new content available on ``mn1.dataone.org`` through :func:`MN_replication.listObjects`. There is no change of state in system metadata resulting from this operation. Step B5. ~~~~~~~~ Copies of ``sciM.1``, ``sysM.1``, and ``sysD.1`` are retrieved to the Coordinating Node, and the system metadata objects are updated on the Coordinating Node. For sake of brevity, only the elements from ReplicationPolicy onwards are shown in the remaining state tables since the remainder of the content does not change. =========================== ========================================== ========================================== Field sysD.1 sysM.1 =========================== ========================================== ========================================== identifier sciD.1 sciM.1 ... replicationPolicy .preferredMemberNode mn2.dataone.org mn2.dataone.org .blockedMemberNode .replicationAllowed true true .numberReplicas 1 1 dateUploaded 2010-03-04T18:13:51.0Z 2010-03-04T18:14:45.0Z dateSysMetadataModified **2010-03-04T19:13:51.0Z** **2010-03-04T19:13:52.0Z** originMemberNode mn1.dataone.org mn1.dataone.org authoritativeMemberNode mn1.dataone.org mn1.dataone.org replica[1] .replicaMemberNode mn1.dataone.org mn1.dataone.org .replicationStatus **Completed** **Completed** .replicaVerified **2010-03-04T19:13:51.0Z** **2010-03-04T19:13:52.0Z** =========================== ========================================== ========================================== Step B6. ~~~~~~~~ The Coordinating Node initiates replication of the objects ``sciM.1`` and ``sciD.1`` from ``mn1.dataone.org`` to ``mn2.dataone.org``. It does this by calling the "replicate()" method on ``mn2.dataone.org``. =========================== ========================================== ========================================== Field sysD.1 sysM.1 =========================== ========================================== ========================================== identifier sciD.1 sciM.1 ... replicationPolicy .preferredMemberNode mn2.dataone.org mn2.dataone.org .blockedMemberNode .replicationAllowed true true .numberReplicas 1 1 dateUploaded 2010-03-04T18:13:51.0Z 2010-03-04T18:14:45.0Z dateSysMetadataModified **2010-03-04T20:58:00.0Z** **2010-03-04T20:58:00.0Z** originMemberNode mn1.dataone.org mn1.dataone.org authoritativeMemberNode mn1.dataone.org mn1.dataone.org replica[1] .replicaMemberNode mn1.dataone.org mn1.dataone.org .replicationStatus Completed Completed .replicaVerified 2010-03-04T19:13:51.0Z 2010-03-04T19:13:52.0Z replica[2] .replicaMemberNode **mn2.dataone.org** **mn2.dataone.org** .replicationStatus **Queued** **Queued** .replicaVerified =========================== ========================================== ========================================== Step B7. ~~~~~~~~ The node ``mn2.dataone.org`` requests the objects ``sciM.1`` and ``sciD.1`` from ``mn1.dataone.org``. After verifying the request with a Coordinating Node (which has a side effect of setting the system metadata ReplicationStatus to Requested), ``mn1.dataone.org`` returns the objects. (Note that in actuality, each object is requested individually.) =========================== ========================================== ========================================== Field sysD.1 sysM.1 =========================== ========================================== ========================================== identifier sciD.1 sciM.1 ... replicationPolicy .preferredMemberNode mn2.dataone.org mn2.dataone.org .blockedMemberNode .replicationAllowed true true .numberReplicas 1 1 dateUploaded 2010-03-04T18:13:51.0Z 2010-03-04T18:14:45.0Z dateSysMetadataModified **2010-03-04T20:59:00.0Z** **2010-03-04T20:59:00.0Z** originMemberNode mn1.dataone.org mn1.dataone.org authoritativeMemberNode mn1.dataone.org mn1.dataone.org replica[1] .replicaMemberNode mn1.dataone.org mn1.dataone.org .replicationStatus Completed Completed .replicaVerified 2010-03-04T19:13:51.0Z 2010-03-04T19:13:52.0Z replica[2] .replicaMemberNode **mn2.dataone.org** **mn2.dataone.org** .replicationStatus **Requested** **Requested** .replicaVerified =========================== ========================================== ========================================== Step B8. ~~~~~~~~ The node ``mn2.dataone.org`` verifies the retrieved objects against their checksums and indicates to the Coordinating Node that transfer was successful. =========================== ========================================== ========================================== Field sysD.1 sysM.1 =========================== ========================================== ========================================== identifier sciD.1 sciM.1 ... replicationPolicy .preferredMemberNode mn2.dataone.org mn2.dataone.org .blockedMemberNode .replicationAllowed true true .numberReplicas 1 1 dateUploaded 2010-03-04T18:13:51.0Z 2010-03-04T18:14:45.0Z dateSysMetadataModified **2010-03-04T21:00:00.0Z** **2010-03-04T21:00:00.0Z** originMemberNode mn1.dataone.org mn1.dataone.org authoritativeMemberNode mn1.dataone.org mn1.dataone.org replica[1] .replicaMemberNode mn1.dataone.org mn1.dataone.org .replicationStatus Completed Completed .replicaVerified 2010-03-04T19:13:51.0Z 2010-03-04T19:13:52.0Z replica[2] .replicaMemberNode mn2.dataone.org mn2.dataone.org .replicationStatus **Completed** **Completed** .replicaVerified **2010-03-04T21:00:00.0Z** **2010-03-04T21:00:00.0Z** =========================== ========================================== ========================================== Final State of System Metadata ------------------------------ Assuming the replication policy was for one replica to be created for ``sciM.1`` and ``sciD.1``, then the eventual end state of the associated system metadata documents (``sysM.1`` and ``sysD.1`` respectively) after synchronization of the Coordinating Nodes, would be something like the following. Note that synchronization between Coordinating Nodes does not trigger an update to the DataSysMetadataModified field. This is necessary to avoid a situation where the state of the metadata is never consistent between Coordinating Nodes. Note also that even though the replication policy of ``sciD.1`` indicates that NumberReplicas is one, there are actually three additional replicas stored on the Coordinating Nodes. =========================== ========================================== ========================================== Field sysD.1 sysM.1 =========================== ========================================== ========================================== identifier sciD.1 sciM.1 objectFormat text/csv eml://ecoinformatics.org/eml-2.1.0 size 1450238 8132 submitter uid=jones,o=NCEAS,dc=ecoinformatics,dc=org uid=jones,o=NCEAS,dc=ecoinformatics,dc=org rightsHolder uid=jones,o=NCEAS,dc=ecoinformatics,dc=org uid=jones,o=NCEAS,dc=ecoinformatics,dc=org obsoletes obsoletedBy derivedFrom describes sciD.1 describedBy sciM.1 checksum 2e01e17467891f7c933dbaa00e1459d23db3fe4f 9967467891f7c933dbaa00e1459d23db3f342 checksumAlgorithm SHA-1 SHA-1 accessRule[1] .ruleType Allow Allow .service Read Read .principal \* \* accessRule[2] .ruleType Allow Allow .service Write Write .principal o=NCEAS,dc=ecoinformatics,dc=org o=NCEAS,dc=ecoinformatics,dc=org replicationPolicy .preferredMemberNode mn2.dataone.org mn2.dataone.org .blockedMemberNode .replicationAllowed true true .numberReplicas 1 1 dateUploaded 2010-03-04T18:13:51.0Z 2010-03-04T18:14:45.0Z dateSysMetadataModified 2010-03-04T18:13:51.0Z 2010-03-04T18:14:45.0Z originMemberNode mn1.dataone.org mn1.dataone.org authoritativeMemberNode mn1.dataone.org mn1.dataone.org replica[1] .replicaMemberNode mn1.dataone.org mn1.dataone.org .replicationStatus Completed Completed .replicaVerified 2010-03-04T19:13:51.0Z 2010-03-04T19:13:52.0Z replica[2] .replicaMemberNode mn2.dataone.org mn2.dataone.org .replicationStatus Completed Completed .replicaVerified 2010-03-04T21:00:00.0Z 2010-03-04T21:00:00.0Z =========================== ========================================== ========================================== Conclusions ----------- Maintaining complete copies of system metadata on all nodes is cumbersome and seems to be overly complicated. However, it is still necessary for Member Nodes to provide accurate information for some elements of the system metadata (to assist with transfer verification, to validate a data object is intact). As such, it seems that scenario B above is the most tractable and provides the desired functionality for tracking content in the DataONE infrastructure. The role of system metadata on Member Nodes and Coordinating Nodes is further clarified below. The recommendations are: - Complete, authoritative copies of system metadata are maintained on the Coordinating Nodes. - A system metadata record becomes authoritative once the replicationStatus is set to complete after initial transfer of content to the Coordinating Node (step B5 above) - Member Nodes still provide access to system metadata through the :func:`MN_crud.getSystemMetadata` calls, though the copy of system metadata returned is authoritative for the properties controlled by the Member Nodes only. - Member Nodes should not return any of the :attr:`SystemMetdata.replica` attributes.