Warning: These documents are under active development and subject to change (version 2.1.0-beta).
The latest release documents are at: https://purl.dataone.org/architecture

Change Log

2014-02-12: Version 1.2. Clarification edits

  • Many edits to clarify definitions such as “archive”.
  • Revised introductory text to APIs and the overall system

2013-10-17: Version 1.2. Minor edits

  • Added language indicating restricted access to log information by public user
  • New version of plantuml.jar

2012-10-09: Version 1.1. Clarifying count parameter for slicing

  • No functional changes were introduced.
  • listObjects, listSubjects, getLogRecords descriptions were updated

2012-09-28: Version 1.1. Adding MNQuery API

  • Added MNQuery API as an optional Tier 1 API with the same method signatures as the CNRead.Query* method definitions.

2012-09-25: Version 1.1. Revising proposed Query API

  • Updated proposed search API following review and discussion. Including draft changes to 1.1 dataoneTypes.xsd
  • Changed search2 to query
  • Changed listSearchFields to getQueryEngineDescription
  • Changed listSearchEngines to listQueryEngines
  • 2012-09-06: Version 1.0. Removed ambiguous exception serialization info

2012-08-20: Version 1.1. Augmenting Search API

DRAFT edits for review:

  • Added method search2 which enables return of arbitrary structure as determined by the selected search engine.
  • Added method listSearchFields which returns a list of fields that can be searched using a specified search engine.
  • Added method listSearchEngines which returns a list of search engines available.

2012-05-31: Version 1.0, Correction to URIs in Resource Maps

  • Corrected URI constructs in the resource map data package documents to use resolve rather than getObject URIs so that clients can take advantage of the ObjectLocationList or HTTP status 302 redirect.

2012-05-02: Version 1.0, MethodCrossReference.xls

  • Moved MN systemMetadataChanged from MNStorage API to MNAuthorization API so that the method appears in Tier 2 operations.

2012-05-01: Version 1.0, MethodCrossReference.xls

  • The MN method getReplica was incorrectly located in Tier 4, effectively preventing any MN implementation except Tier 4 nodes from acting as replication sources. This has been corrected, with the method moved to the MNRead Tier1 API. The method description was updated accordingly.
  • Existing MNStorage.delete renamed to MNStorage.archive, better reflecting its action.
  • New MNStorage.delete operation added for Tier3. This operation does actually delete the content from the node, removing it from DataONE services (the node may choose to also delete the bytes)
  • New CNCore.archive operation performs the same action as MN.archive, except the operation triggers a change notification to all nodes containing the object.
  • New CNCore.delete operation intended to be used only by administrators when there is a requirement to remove an object from the entire DataONE system. Memory of the identifier persists, preventing its reuse.

2012-03-26: Version 1.0, MethodCrossReference.xls

  • Corrected typo in REST description for listObjects which had lowercase formatid, corrected to formatId.
  • Corrected order of parameters for getLogRecords, moving start and count to the end to be consistent with other method signatures.

2012-03-26: Version 1.0, MethodCrossReference.xls

2012-03-25: Version 1.0, MethodCrossReference.xls

  • MNCore.getLogRecords() added an optional parameter pid that specifies the prefix of the pid for matching on log records. Support for this parameter is optional, and a MN implementation may silently ignore if present in the request and the node is unable to support the pid prefix filter.
  • CNCore.getLogRecords() added an optional parameter pid that specifies the prefix of the pid for matching on log records. Support for this parameter is currently optional for CNs, and may be ignored if present in the request.
  • MNStorage.GenerateIdentifier() new method added to the MNStorage API. This signature mirrors the CNCore.GenerateIdentifier method and is made available on Member Nodes as they may choose to have an organizational association with an identifier service.

2012-02-27: Version 1.0, MethodCrossReference.xls

  • CNRead.search() clarified how to specify path and query separators in the value of parameter ‘query.’ Change xmit type of parameter ‘query’ to ‘path’, since it contains both path and query (to simplify client conformity testing)

2012-02-24 : dataoneTypes.xsd

  • Correction to the contrabEntry definition
  • Several documentation only changes

2012-02-17 : Version 1.0, MethodCrossReference.xls

  • CNRead.search() clarified that queryType is specified as a URL path parameter and the remainder of the URL (path and query) is passed to the respective search engine as indicated by the value of queryType.

2012-02-15 : Version 1.0, MethodCrossReference.xls

2012-02-02 : Version 1.0, MethodCrossReference.xls

2012-02-01 : Version 1.0, MethodCrossReference.xls

2012-01-20 : Version 1.0, MethodCrossReference.xls

2012-01-19 : Version 1.0, MethodCrossReference.xls

2012-01-12 : Version 1.0, MethodCrossReference.xls

2012-01-10 : Version 1.0, MethodCrossReference.xls

  • updated method descriptions for updateReplicaMetadata
  • updated method descriptions for updateNodeCapabilities
  • removed VersionMismatch from setReplicationStatus method

2012-01-03 : Version 1.0, MethodCrossReference.xls

  • removed the deprecated assertRelation method

2011-12-19 : Version 1.0, dataoneTypes.xsd

  • replaced tabs with ” “

  • Event names in description changed to same case as enumerated values

  • various: minor formatting changes for rendering in architecture docs.

  • Added root elements for AccessRule, LogEntry, NodeReplicationPolicy, ObjectInfo, Service, Services, Schedule, Synchronization, NodeReplicationPolicy, ServiceMethodRestriction.

    AccessRule, LogEntry, NodeReplicationPolicy, ObjectInfo, Service, Services, Schedule, Synchronization, NodeReplicationPolicy, ServiceMethodRestriction.

2011-12-19 : Version 1.0

  • Altered MNCore and CNCore ping() to use the HTTP Date header field for transmitting the timestamp. Removed requirement for timestamp appearing in response body.

2011-12-14 : Version 1.0

  • Changed MNCore.ping() to return a DateTime string
  • Added CNCore.ping()

2011-12-13 : Version 1.0

  • Updated MethodCrossReference.xls to reflect API changes for CN.setReplicationStatus()
  • Updated MethodCrossReference.xls to reflect API changes for CN.isNodeAuthorized()

2011-12-12 : Version 1.0

Changes to dataoneTypes.xsd:

  • Added ChecksumAlgorithmList
  • Limited valid checksums to SHA-1 and MD5
  • Added ReplicationStatus.FAILED to the enumeration
  • Added a new SimpleType: NonEmptyNoWhitespaceString800, used to restrict identifiers
  • Removed Permission.REPLICATE
  • Changed SystemMetadata to make the following fields optional: submitter, dateUploaded, dateSystemMetadataModified, serialVersion. However, an MN or CN must set them.
  • Created Types.CrontabEntrySeconds to restrict the seconds field in a Schedule
  • Changed documentation for SystemMetadata.authoritativeMemberNode
  • Added a new NodeReplicationPolicy type, to be used as an optional structure in Node
  • Removed MonitorInfo and MonitorList types - deprecated.
  • Updated documentation for the Node type.

MN API

  • MNCore.getLogRecords and MNRead.listObjects - the time comparison range has been changed. The upper bound is now exclusive, i.e. fromDate <= date < toDate
  • MNRead.listObjects - the name of the date range comparison properties has been changed from “startTime” and “endTime” to “fromDate” and “toDate” respectively.
  • MNRead.get - added InsufficientResources exception as possible response. This may be raised for example, if memory, CPU, or bandwidth use is too limited to respond to the request.
  • MNCore.ping - added description for why InsufficientResources may be raised.

CN API

  • CNCore.getLogRecords and CNRead.listObjects - the time comparison range has been changed. The upper bound is now exclusive, i.e. fromDate <= date < toDate
  • CNRead.listObjects - the name of the date range comparison properties has been changed from “startTime” and “endTime” to “fromDate” and “toDate” respectively.
  • CNIdentity.getSubjectInfo, added InvalidToken exception
  • CNRegister.updateNodeCapabilities, added InvalidToken exception
  • CNReplication.updateReplicationMetadata, added InvalidToken exception
  • CNCore.listFormats: removed InsufficientResources exception. If the CN can not provide a response, then a ServiceFailure should be raised.
  • CNCore.getChecksumAlgorithms renamed to listChecksumAlgorithms for consistency with other methods that return a list.

2011-12-08 : Version 1.0

MethodCrossReference.xls (trunk)

  • CN.removeGroupMembers - changed REST specification from “DELETE /groups/{group}” to “POST /groups/remove/{group}
  • CN.getFormat: removed InvalidRequest from documentation, (was supposedly removed earlier, but reappeared)
  • CN.listSubjects - added status parameter to the method
  • CN.search - added queryType and query parameters as optional (documentation fix)
  • changed setOwner() to setRightsHolder()
  • MN.getReplica - documentation fix, added pid as parameter - it was in REST specification but not listed as a parameter

2011-12-07-api-corrections branch

Java libclient / implementation changes (in branch)

  • CN.verifyAccount - removed NotFound from java api and impl
  • CN.updateAccount - removed IdentifierNotUnique from java api and impl
  • CN.mapIdentity - refactored method signature from (session, subject1, subject2) to (session,subject)
  • CN.mapIdentity - changed message body to pass subject as paramPart instead of filePart.
  • CN.requestMapIdentity - added IdentifierNotUnique to java api and impl
  • CN.denyMapIdentity - removed InvalidRequest from java api and impl
  • CN.removeMapIdentity - removed InvalidRequest from java api and impl
  • CN.search - changed resource in java libclient from ‘object’ to ‘search’
  • CN.setAccessPolicy - put parameter ‘pid’ on path (removed from filePart)
  • CN.setOwner - changed resource from ‘accounts/map’ to ‘owner’
  • MN.getReplica - added InsufficientResources to java api and impl
  • MN.setAccessPolicy - removed method from java api and impl
  • CN.setOwner - changed name to setRightsHolder in java api and impl

Client bug fixes (in trunk)

  • MN.getLogRecords - added missing ‘start’ and ‘count’ parameters to impl call.
  • MN.listObjects - fixed failed exception recast

2011-12-06 Version 1.0.0 Branch

dataoneTypes.xsd

2011-12-05 Version 1.0.0 Branch

dataoneTypes.xsd

2011-11-08 Version 1.0.0 Branch

CN APIs

  • CNCore.hasReservation: Updated description: caller owns reservation = HTTP 200; PID reserved but not owned by caller = NotAuthorized; PID in use as an identifier = IdentifierNotUnique; PID not reserved and not in use = NotFound.
  • CNRead.updateSystemMetadata: DEPRECATED. There is no longer a need for this method to be exposed through an external interface.

2011-11-03 Version 1.0.0 Branch

MN APIs

  • MNCore.getLogRecords: change date restriction to >= from Date, and <= toDate
  • MNCore.getCapabilities: removed NotAuthorized exception. Anyone should be able to call this method.
  • MNRead.listObjects: Indicated that if formatId is not present as a filter parameter, then that filter should be ignored. The previous was to default to true.

2011-11-02 Version 1.0.0 Branch

dataoneTypes.xsd

  • https://redmine.dataone.org/projects/d1/repository/revisions/5754
  • Deprecated MNAuthorization.setAccessPolicy. This is replaced by MNStorage.systemMetadataChanged
  • Added MNStorage.systemMetadataChanged, which is called by Coordinating Nodes when system metadata for an object known to be located on the Member Node changes. The Member Node is expected to retrieve the full system metadata document from the Coordinating Node and update the local system properties of the object using information contained in the authoritative system metadata. This is a Tier 2 method. It is accessed on the /dirtySystemMetadata REST endpoint.

2011-11-01 Version 1.0.0 Branch

Exceptions

  • Added a new exception, VersionMistmatch which is raised when there is a version mistmatch between the target of the request and the supplied parameters. e.g. when serialVersion is provided in a call and does not match that of the target.

CN APIs

  • CNAuthorization.setOwner: added serialVersion parameter to method signature. serialVersion is used to indicate which revision the change to system metadata applies to. If the revision in the request does not match the revision in the response, then a VersionMistmatch exception is raised.
  • CNAuthorization.getAccessPolicy: added serialVersion parameter to method signature.
  • CNReplication.setReplicationStatus: added serialVersion parameter to method signature.
  • CNReplication.updateReplicationMetadata: added serialVersion parameter to method signature.
  • CNReplication.setReplicationPolicy: added serialVersion parameter to method signature.

MN APIs

  • MNCore.getLogRecords: changed fromDate from required to optional.
  • MNCore.getCapabilities: removed InvalidRequest exception
  • MNCore.get: removed InvalidRequest exception
  • MNRead.getSystemMetadata: removed InvalidRequest exception
  • MNRead.describe: removed InvalidRequest exception
  • MNRead.synchronizationFailed: removed InvalidRequest exception
  • MNAuthorization.setAccessPolicy: Added note indicating that this method should only be called by Coordinating Nodes.
  • MNStorage.delete: removed InvalidRequest exception

2011-10-26 Version 1.0.0 Branch

dataoneTypes.xsd

The following changes were enacted on the dataoneTypes.xsd schema as a result of discussions during the DataONE all-hands meeting held during the week of October 18, 2011.

No further significant changes to dataonetypes.xsd are expected for version 1.0.0 of the DataONE infrastructure.

  • unsigned long “serialversion” element added to the Types.SystemMetadata complex type. The serialVersion value is incremented upon any updates to a system metadata instance, and is used to indicate which is the current version (highest serialVersion is always the latest) of system metadata for an object.
  • “fmtid” change to “formatId” in Types.SystemMetadata complex type
  • “fmtid” change to “formatId” in Types.ObjectFormat complex type
  • “fmtid” change to “formatId” in Types.ObjectInfo complex type
  • email minOccurs change from 1 to 0 in Types.Person complex type
  • boolean “verified” property added to Types.Person complex type. This value is true if the name and email address of the Person have been verified to ensure that the givenName and familyName represent the real person’s legal name, and that the email address is correct for that person and is in the control of the indicated individual. Verification occurs through a established procedure within DataONE as part of the Identity Management system.
  • Numerous annotation edits and updates

CN APIs

  • CNCore.listFormats: removed InvalidRequest and NotFound exceptions
  • CNCore.getFormat: removed InvalidRequest exception
  • CNCore.getLogRecords:
    • Made fromDate optional. When not provided in request, there is no limit to the earliest record being returned.
    • Added exception InsufficientResources (443, 1481)
  • CNCore.listNodes: removed InvalidRequest exception
  • CNCore.hasReservation: removed exception IdentifierNotUnique
  • CNRead.get: removed InvalidRequest exception
  • CNRead.getSystemMetadata: removed InvalidRequest exception
  • CNRead.resolve: removed InvalidRequest exception
  • CNRead.assertRelation : deprecated
  • CNRead.getChecksum: removed InvalidRequest exception
  • CNIdentity.registerAccount: added NotAuthorized exception, thrown when subject of session does not match that of the person.
  • CNIdentity.updateAccount:
    • added exception NotAuthorized, raised if subject of session does not match that of the person.
    • added exception NotFound, raised if the account does not exist.
  • CNIdentity.verifyAccount:
    • changed HTTP method to PUT (this is an update operation)
    • Added exception NotFound, raised if the specified account does not already exist.
  • CNIdentity.getSubjectInfo:
    • added exception NotFound, raised if the requested subject is not registered.
    • removed exception InvalidRequest
  • CNIdentity.listSubjects
    • Specified default start value of 0
    • Specified default page size (count) of 100
    • Added InvalidRequest exception
  • CNIdentity.mapIdentity: added exception IdentfierNotUnique, raised when the subject of the session and the provided subject arethe same.
  • CNIdentity.confirmMapIdentity
    • changed REST URL to be a PUT to /accounts/map/{subject}
    • Removed the InvalidRequest exception
    • Updated the description of the NotFound exception
  • CNIdentity.createGroup
    • Removed NotFound exception
    • Removed InvalidRequest exception
  • CNReplication.setReplicationStatus: corrected description of the REST method signature. Parameters for the PUT request should be transmitted in the request body, the session information should be transmitted via SSL handshake process.
  • CNReplication.updateReplicationStatus:
    • corrected REST sURL description as for setReplicationStatus
    • Added NotFound exception
  • CNReplication.setReplicationPolicy: updated REST URL description as per setReplicationStatus.
  • CNReplication.isNodeAuthorized: updated REST URL description as per setReplicationStatus

This material is based upon work supported by the National Science Foundation under Grant Numbers 083094 and 1430508.

Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.