Äcdocutils.nodes
document
q)Åq}q(U	nametypesq}q(X���goalqNX���triggersqNX���implementationqNX���use case 06 - mn synchronizeq	NX���in more detailq
NX���summaryqNX���implementation overviewqNX���implementation detailsq
NX
���preconditionsqNX���post conditionsqNX���actorsqNX���uc06qàX���historyqàuUsubstitution_defsq}qUparse_messagesq]qUcurrent_sourceqNU
decorationqNUautofootnote_startqKUnameidsq}q(hUgoalqhUtriggersqhUimplementationqh	Uuse-case-06-mn-synchronizeqh
Uin-more-detailq hUsummaryq!hUimplementation-overviewq"h
Uimplementation-detailsq#hU
preconditionsq$hUpost-conditionsq%hUactorsq&hUuc06q'hUhistoryq(uUchildrenq)]q*(cdocutils.nodes
target
q+)Åq,}q-(U	rawsourceq.X	���.. _UC06:Uparentq/hUsourceq0Xj���/var/lib/jenkins/jobs/API_Documentation_trunk/workspace/api-documentation/source/design/UseCases/06_uc.txtq1Utagnameq2Utargetq3U
attributesq4}q5(Uidsq6]Ubackrefsq7]Udupnamesq8]Uclassesq9]Unamesq:]Urefidq;h'uUlineq<KUdocumentq=hh)]ubcdocutils.nodes
section
q>)Åq?}q@(h.U�h/hh0h1Uexpect_referenced_by_nameqA}qBhh,sh2UsectionqCh4}qD(h8]h9]h7]h6]qE(hh'eh:]qF(h	heuh<Kh=hUexpect_referenced_by_idqG}qHh'h,sh)]qI(cdocutils.nodes
title
qJ)ÅqK}qL(h.X���Use Case 06 - MN SynchronizeqMh/h?h0h1h2UtitleqNh4}qO(h8]h9]h7]h6]h:]uh<Kh=hh)]qPcdocutils.nodes
Text
qQX���Use Case 06 - MN SynchronizeqRÖÅqS}qT(h.hMh/hKubaubcsphinx.addnodes
index
qU)ÅqV}qW(h.U�h/h?h0h1h2UindexqXh4}qY(h6]h7]h8]h9]h:]UentriesqZ]q[((Usingleq\X���Use Case 06Uindex-0q]U�Ntq^(h\X���UC06h]U�Ntq_(h\X���MN Synchronizeh]U�Ntq`(h\X���synchronizeh]U�NtqaeUinlineqbâuh<Kh=hh)]ubh+)Åqc}qd(h.U�h/h?h0h1h2h3h4}qe(h6]h7]h8]h9]h:]h;h]uh<Kh=hh)]ubh>)Åqf}qg(h.U�h/h?h0h1hA}h2hCh4}qh(h8]h9]h7]h6]qi(hh]eh:]qjhauh<K	h=hhG}qkh]hcsh)]ql(hJ)Åqm}qn(h.X���Goalqoh/hfh0h1h2hNh4}qp(h8]h9]h7]h6]h:]uh<K	h=hh)]qqhQX���GoalqrÖÅqs}qt(h.hoh/hmubaubcdocutils.nodes
paragraph
qu)Åqv}qw(h.Xê���Science metadata records, resource maps, and system metadata for all objects
available on a Member Node are synchronized to a Coordinating Node.qxh/hfh0h1h2U	paragraphqyh4}qz(h8]h9]h7]h6]h:]uh<Kh=hh)]q{hQX��Science metadata records, resource maps, and system metadata for all objects
available on a Member Node are synchronized to a Coordinating Node.q|ÖÅq}}q~(h.hxh/hvubaubeubh>)Åq}qÄ(h.U�h/h?h0h1h2hCh4}qÅ(h8]h9]h7]h6]qÇh!ah:]qÉhauh<Kh=hh)]qÑ(hJ)ÅqÖ}qÜ(h.X���Summaryqáh/hh0h1h2hNh4}qà(h8]h9]h7]h6]h:]uh<Kh=hh)]qâhQX���SummaryqäÖÅqã}qå(h.háh/hÖubaubhu)Åqç}qé(h.X‚��As content is added to Member Nodes, the associated system metadata, science
metadata, and resource map documents are retrieved by a Coordinating Node.
Once the objects are registered with the Coordinating Node, they are
considered to be part of the DataONE system, and so will be managed
accordingly - the metadata and resource map object will be replicated between
Coordinating Nodes, metadata will be indexed, and the objects will be
replicated to other Member Nodes as required.qèh/hh0h1h2hyh4}qê(h8]h9]h7]h6]h:]uh<Kh=hh)]qëhQX‚��As content is added to Member Nodes, the associated system metadata, science
metadata, and resource map documents are retrieved by a Coordinating Node.
Once the objects are registered with the Coordinating Node, they are
considered to be part of the DataONE system, and so will be managed
accordingly - the metadata and resource map object will be replicated between
Coordinating Nodes, metadata will be indexed, and the objects will be
replicated to other Member Nodes as required.qíÖÅqì}qî(h.hèh/hçubaubhu)Åqï}qñ(h.XQ���Content replication as managed by Coordinating Nodes is described in Use Case
09.qóh/hh0h1h2hyh4}qò(h8]h9]h7]h6]h:]uh<Kh=hh)]qôhQXQ���Content replication as managed by Coordinating Nodes is described in Use Case
09.qöÖÅqõ}qú(h.hóh/hïubaubhu)Åqù}qû(h.Xı���In version 1.x of the DataONE infrastructure, synchronization is through polling
only. A Coordinating Node will periodically request a list of objects that are
new or have been altered for some specified time period using the listObjects
method.qüh/hh0h1h2hyh4}q†(h8]h9]h7]h6]h:]uh<Kh=hh)]q°hQXı���In version 1.x of the DataONE infrastructure, synchronization is through polling
only. A Coordinating Node will periodically request a list of objects that are
new or have been altered for some specified time period using the listObjects
method.q¢ÖÅq£}q§(h.hüh/hùubaubhu)Åq•}q¶(h.X��In version 2.x of the DataONE infrastructure, a new mechanism to enable clients
or Member Nodes to request synchronization of an object was added. This helps
ensure a more rapid update of object properties such as access control rules are
propogated through the infrastructure.qßh/hh0h1h2hyh4}q®(h8]h9]h7]h6]h:]uh<K!h=hh)]q©hQX��In version 2.x of the DataONE infrastructure, a new mechanism to enable clients
or Member Nodes to request synchronization of an object was added. This helps
ensure a more rapid update of object properties such as access control rules are
propogated through the infrastructure.q™ÖÅq´}q¨(h.hßh/h•ubaubeubh>)Åq≠}qÆ(h.U�h/h?h0h1h2hCh4}qØ(h8]h9]h7]h6]q∞h&ah:]q±hauh<K(h=hh)]q≤(hJ)Åq≥}q¥(h.X���Actorsqµh/h≠h0h1h2hNh4}q∂(h8]h9]h7]h6]h:]uh<K(h=hh)]q∑hQX���Actorsq∏ÖÅqπ}q∫(h.hµh/h≥ubaubcdocutils.nodes
bullet_list
qª)Åqº}qΩ(h.U�h/h≠h0h1h2Ubullet_listqæh4}qø(Ubulletq¿X���-h6]h7]h8]h9]h:]uh<K*h=hh)]q¡(cdocutils.nodes
list_item
q¬)Åq√}qƒ(h.X���Member Nodeq≈h/hºh0h1h2U	list_itemq∆h4}q«(h8]h9]h7]h6]h:]uh<Nh=hh)]q»hu)Åq…}q (h.h≈h/h√h0h1h2hyh4}qÀ(h8]h9]h7]h6]h:]uh<K*h)]qÃhQX���Member NodeqÕÖÅqŒ}qœ(h.h≈h/h…ubaubaubh¬)Åq–}q—(h.X���Coordinating Node
h/hºh0h1h2h∆h4}q“(h8]h9]h7]h6]h:]uh<Nh=hh)]q”hu)Åq‘}q’(h.X���Coordinating Nodeq÷h/h–h0h1h2hyh4}q◊(h8]h9]h7]h6]h:]uh<K+h)]qÿhQX���Coordinating NodeqŸÖÅq⁄}q€(h.h÷h/h‘ubaubaubeubcsphinxcontrib.plantuml
plantuml
q‹)Åq›}qfi(h.XS��.. uml::

   @startuml images/06_uc.png
   actor "Coordinating Node" as CN
   actor "Member Node" as MN

   usecase "13. Authorization" as author
   usecase "06. Synchronize Metadata" as SYNC
   usecase "43. Notify Indexer" as NOTIFY
   CN -- SYNC
   MN -- SYNC
   SYNC ..> author: <<includes>>
   SYNC ..> NOTIFY: <<includes>>
   @enduml
h/h≠h0h1h2Uplantumlqflh4}q‡(h6]h7]h8]h9]h:]Uumlq·X'��@startuml images/06_uc.png
actor "Coordinating Node" as CN
actor "Member Node" as MN

usecase "13. Authorization" as author
usecase "06. Synchronize Metadata" as SYNC
usecase "43. Notify Indexer" as NOTIFY
CN -- SYNC
MN -- SYNC
SYNC ..> author: <<includes>>
SYNC ..> NOTIFY: <<includes>>
@endumluh<K;h=hh)]ubhu)Åq‚}q„(h.Xg���**Figure 1.** Use case diagram indicating the components and other use cases
involved in this use case.h/h≠h0h1h2hyh4}q‰(h8]h9]h7]h6]h:]uh<K<h=hh)]qÂ(cdocutils.nodes
strong
qÊ)ÅqÁ}qË(h.X
���**Figure 1.**h4}qÈ(h8]h9]h7]h6]h:]uh/h‚h)]qÍhQX	���Figure 1.qÎÖÅqÏ}qÌ(h.U�h/hÁubah2UstrongqÓubhQXZ��� Use case diagram indicating the components and other use cases
involved in this use case.qÔÖÅq}qÒ(h.XZ��� Use case diagram indicating the components and other use cases
involved in this use case.h/h‚ubeubeubh>)ÅqÚ}qÛ(h.U�h/h?h0h1h2hCh4}qÙ(h8]h9]h7]h6]qıh$ah:]qˆhauh<KAh=hh)]q˜(hJ)Åq¯}q˘(h.X
���Preconditionsq˙h/hÚh0h1h2hNh4}q˚(h8]h9]h7]h6]h:]uh<KAh=hh)]q¸hQX
���Preconditionsq˝ÖÅq˛}qˇ(h.h˙h/h¯ubaubhª)År���}r��(h.U�h/hÚh0h1h2hæh4}r��(h¿X���-h6]h7]h8]h9]h:]uh<KCh=hh)]r��(h¬)År��}r��(h.X���CN and MN operationalr��h/j���h0h1h2h∆h4}r��(h8]h9]h7]h6]h:]uh<Nh=hh)]r��hu)År	��}r
��(h.j��h/j��h0h1h2hyh4}r��(h8]h9]h7]h6]h:]uh<KCh)]r��hQX���CN and MN operationalr
��ÖÅr��}r��(h.j��h/j	��ubaubaubh¬)År��}r��(h.X)���New content available on a Member Node.

h/j���h0h1h2h∆h4}r��(h8]h9]h7]h6]h:]uh<Nh=hh)]r��hu)År��}r��(h.X'���New content available on a Member Node.r��h/j��h0h1h2hyh4}r��(h8]h9]h7]h6]h:]uh<KDh)]r��hQX'���New content available on a Member Node.r��ÖÅr��}r��(h.j��h/j��ubaubaubeubeubh>)År��}r��(h.U�h/h?h0h1h2hCh4}r��(h8]h9]h7]h6]r��hah:]r ��hauh<KHh=hh)]r!��(hJ)År"��}r#��(h.X���Triggersr$��h/j��h0h1h2hNh4}r%��(h8]h9]h7]h6]h:]uh<KHh=hh)]r&��hQX���Triggersr'��ÖÅr(��}r)��(h.j$��h/j"��ubaubhª)År*��}r+��(h.U�h/j��h0h1h2hæh4}r,��(h¿X���-h6]h7]h8]h9]h:]uh<KJh=hh)]r-��(h¬)År.��}r/��(h.X���Periodic polling of MNsr0��h/j*��h0h1h2h∆h4}r1��(h8]h9]h7]h6]h:]uh<Nh=hh)]r2��hu)År3��}r4��(h.j0��h/j.��h0h1h2hyh4}r5��(h8]h9]h7]h6]h:]uh<KJh)]r6��hQX���Periodic polling of MNsr7��ÖÅr8��}r9��(h.j0��h/j3��ubaubaubh¬)År:��}r;��(h.X@���Signal to CN issued by a MN that is requesting synchronization

h/j*��h0h1h2h∆h4}r<��(h8]h9]h7]h6]h:]uh<Nh=hh)]r=��hu)År>��}r?��(h.X>���Signal to CN issued by a MN that is requesting synchronizationr@��h/j:��h0h1h2hyh4}rA��(h8]h9]h7]h6]h:]uh<KKh)]rB��hQX>���Signal to CN issued by a MN that is requesting synchronizationrC��ÖÅrD��}rE��(h.j@��h/j>��ubaubaubeubeubh>)ÅrF��}rG��(h.U�h/h?h0h1h2hCh4}rH��(h8]h9]h7]h6]rI��h%ah:]rJ��hauh<KOh=hh)]rK��(hJ)ÅrL��}rM��(h.X���Post ConditionsrN��h/jF��h0h1h2hNh4}rO��(h8]h9]h7]h6]h:]uh<KOh=hh)]rP��hQX���Post ConditionsrQ��ÖÅrR��}rS��(h.jN��h/jL��ubaubhª)ÅrT��}rU��(h.U�h/jF��h0h1h2hæh4}rV��(h¿X���-h6]h7]h8]h9]h:]uh<KQh=hh)]rW��(h¬)ÅrX��}rY��(h.Xg���All System Metadata, science metadata, and resoruce map documents from a MN
are also present on the CN
h/jT��h0h1h2h∆h4}rZ��(h8]h9]h7]h6]h:]uh<Nh=hh)]r[��hu)År\��}r]��(h.Xf���All System Metadata, science metadata, and resoruce map documents from a MN
are also present on the CNr^��h/jX��h0h1h2hyh4}r_��(h8]h9]h7]h6]h:]uh<KQh)]r`��hQXf���All System Metadata, science metadata, and resoruce map documents from a MN
are also present on the CNra��ÖÅrb��}rc��(h.j^��h/j\��ubaubaubh¬)Ård��}re��(h.X7���Update to the search index is triggered (Use Case 43)

h/jT��h0h1h2h∆h4}rf��(h8]h9]h7]h6]h:]uh<Nh=hh)]rg��hu)Årh��}ri��(h.X5���Update to the search index is triggered (Use Case 43)rj��h/jd��h0h1h2hyh4}rk��(h8]h9]h7]h6]h:]uh<KTh)]rl��hQX5���Update to the search index is triggered (Use Case 43)rm��ÖÅrn��}ro��(h.jj��h/jh��ubaubaubeubeubh>)Årp��}rq��(h.U�h/h?h0h1h2hCh4}rr��(h8]h9]h7]h6]rs��h"ah:]rt��hauh<KXh=hh)]ru��(hJ)Årv��}rw��(h.X���Implementation Overviewrx��h/jp��h0h1h2hNh4}ry��(h8]h9]h7]h6]h:]uh<KXh=hh)]rz��hQX���Implementation Overviewr{��ÖÅr|��}r}��(h.jx��h/jv��ubaubh‹)År~��}r��(h.X��.. uml::

  @startuml images/06_uc_a.png
  autonumber "[0] "
  participant MN <<Member Node>>
  participant CN <<Coordinating Node>>

  CN -> MN: listObjects( timePeriod )
  activate CN
  MN -> CN: objectList
  loop "for each PID"
    CN -> CN: queue PID for synchronization
  end
    deactivate CN

  ... <font color=red>**Possibly Lengthy Delay** ...

  CN -> MN: getSystemMetadata(PID)
  activate CN
  activate MN
  MN -> CN: SystemMetadata
  deactivate MN
  alt New Objec
    activate CN
    CN -> CN: store System Metadata
    CN ->o]: Notify index of new System Metadata
    deactivate CN
    alt "Is Science Metadata or Resource Map"
      CN -> MN: get(PID)
      activate CN
      activate MN
      MN -> CN: object
      deactivate MN
      CN -> CN: store object
      CN ->o]: Notify index of new content
      deactivate CN
    end
  else "Existing Object"

    activate CN
    CN -> CN: update system metadata properties
    CN ->o]: Notify index of modified content
    deactivate CN
  end
  deactivate CN
  @enduml
h/jp��h0h1h2hflh4}rÄ��(h6]h7]h8]h9]h:]h·X¨��@startuml images/06_uc_a.png
autonumber "[0] "
participant MN <<Member Node>>
participant CN <<Coordinating Node>>

CN -> MN: listObjects( timePeriod )
activate CN
MN -> CN: objectList
loop "for each PID"
  CN -> CN: queue PID for synchronization
end
  deactivate CN

... <font color=red>**Possibly Lengthy Delay** ...

CN -> MN: getSystemMetadata(PID)
activate CN
activate MN
MN -> CN: SystemMetadata
deactivate MN
alt New Objec
  activate CN
  CN -> CN: store System Metadata
  CN ->o]: Notify index of new System Metadata
  deactivate CN
  alt "Is Science Metadata or Resource Map"
    CN -> MN: get(PID)
    activate CN
    activate MN
    MN -> CN: object
    deactivate MN
    CN -> CN: store object
    CN ->o]: Notify index of new content
    deactivate CN
  end
else "Existing Object"

  activate CN
  CN -> CN: update system metadata properties
  CN ->o]: Notify index of modified content
  deactivate CN
end
deactivate CN
@endumluh<Kàh=hh)]ubhu)ÅrÅ��}rÇ��(h.X£���**Figure 2.** Overview of synchronization process using a polling process to
periodically request information about objects that have changed within a time
period.h/jp��h0h1h2hyh4}rÉ��(h8]h9]h7]h6]h:]uh<Kâh=hh)]rÑ��(hÊ)ÅrÖ��}rÜ��(h.X
���**Figure 2.**h4}rá��(h8]h9]h7]h6]h:]uh/jÅ��h)]rà��hQX	���Figure 2.râ��ÖÅrä��}rã��(h.U�h/jÖ��ubah2hÓubhQXñ��� Overview of synchronization process using a polling process to
periodically request information about objects that have changed within a time
period.rå��ÖÅrç��}ré��(h.Xñ��� Overview of synchronization process using a polling process to
periodically request information about objects that have changed within a time
period.h/jÅ��ubeubh‹)Årè��}rê��(h.X‘��.. uml::

  @startuml images/06_seq_a
  autonumber "[0] "
  participant MN <<Member Node>>
  participant CN <<Coordinating Node>>

  MN -> CN: synchronize(PID)
  activate CN
  CN -> CN: queue PID for synchronization
  CN -> MN: ack
  deactivate CN

  ... <font color=red>**Possibly Lengthy Delay** ...

  CN -> MN: getSystemMetadata(PID)
  activate CN
  activate MN
  MN -> CN: SystemMetadata
  deactivate MN
  alt New Objec
    activate CN
    CN -> CN: store System Metadata
    CN ->o]: Notify index of new System Metadata
    deactivate CN
    alt "Is Science Metadata or Resource Map"
      CN -> MN: get(PID)
      activate CN
      activate MN
      MN -> CN: object
      deactivate MN
      CN -> CN: store object
      CN ->o]: Notify index of new content
      deactivate CN
    end
  else "Existing Object"

    activate CN
    CN -> CN: update system metadata properties
    CN ->o]: Notify index of modified content
    deactivate CN
  end
  deactivate CN
  @enduml
h/jp��h0h1h2hflh4}rë��(h6]h7]h8]h9]h:]h·X}��@startuml images/06_seq_a
autonumber "[0] "
participant MN <<Member Node>>
participant CN <<Coordinating Node>>

MN -> CN: synchronize(PID)
activate CN
CN -> CN: queue PID for synchronization
CN -> MN: ack
deactivate CN

... <font color=red>**Possibly Lengthy Delay** ...

CN -> MN: getSystemMetadata(PID)
activate CN
activate MN
MN -> CN: SystemMetadata
deactivate MN
alt New Objec
  activate CN
  CN -> CN: store System Metadata
  CN ->o]: Notify index of new System Metadata
  deactivate CN
  alt "Is Science Metadata or Resource Map"
    CN -> MN: get(PID)
    activate CN
    activate MN
    MN -> CN: object
    deactivate MN
    CN -> CN: store object
    CN ->o]: Notify index of new content
    deactivate CN
  end
else "Existing Object"

  activate CN
  CN -> CN: update system metadata properties
  CN ->o]: Notify index of modified content
  deactivate CN
end
deactivate CN
@endumluh<K∫h=hh)]ubhu)Årí��}rì��(h.X∞���**Figure 3.** Overview of synchronization process with an external request to
synchronize an object rather than relying on periodic polling. Note this is a
Version 2.x feature.h/jp��h0h1h2hyh4}rî��(h8]h9]h7]h6]h:]uh<Kªh=hh)]rï��(hÊ)Årñ��}ró��(h.X
���**Figure 3.**h4}rò��(h8]h9]h7]h6]h:]uh/jí��h)]rô��hQX	���Figure 3.rö��ÖÅrõ��}rú��(h.U�h/jñ��ubah2hÓubhQX£��� Overview of synchronization process with an external request to
synchronize an object rather than relying on periodic polling. Note this is a
Version 2.x feature.rù��ÖÅrû��}rü��(h.X£��� Overview of synchronization process with an external request to
synchronize an object rather than relying on periodic polling. Note this is a
Version 2.x feature.h/jí��ubeubeubh>)År†��}r°��(h.U�h/h?h0h1h2hCh4}r¢��(h8]h9]h7]h6]r£��h#ah:]r§��h
auh<K¡h=hh)]r•��(hJ)År¶��}rß��(h.X���Implementation Detailsr®��h/j†��h0h1h2hNh4}r©��(h8]h9]h7]h6]h:]uh<K¡h=hh)]r™��hQX���Implementation Detailsr´��ÖÅr¨��}r≠��(h.j®��h/j¶��ubaubh+)ÅrÆ��}rØ��(h.U�h/j†��h0h1h2h3h4}r∞��(h6]h7]h8]h9]h:]h;Uindex-1r±��uh<Nh=hh)]ubcsphinx.ext.todo
todo_node
r≤��)År≥��}r¥��(h.X!���2015-09-17 Review for currentnessrµ��h/j†��h0h1hA}h2U	todo_noder∂��h4}r∑��(h8]h9]r∏��Uadmonition-todorπ��ah7]h6]r∫��j±��ah:]uh<K√h=hhG}rª��j±��jÆ��sh)]rº��(hJ)ÅrΩ��}ræ��(h.X���Todorø��h4}r¿��(h8]h9]h7]h6]h:]uh/j≥��h)]r¡��hQX���Todor¬��ÖÅr√��}rƒ��(h.U�h/jΩ��ubah2hNubhu)År≈��}r∆��(h.jµ��h/j≥��h0h1h2hyh4}r«��(h8]h9]h7]h6]h:]uh<K√h)]r»��hQX!���2015-09-17 Review for currentnessr…��ÖÅr ��}rÀ��(h.jµ��h/j≈��ubaubeubcdocutils.nodes
image
rÃ��)ÅrÕ��}rŒ��(h.X���.. image:: images/06_seq.png
h/j†��h0h1h2Uimagerœ��h4}r–��(UuriX!���design/UseCases/images/06_seq.pngr—��h6]h7]h8]h9]U
candidatesr“��}r”��U*j—��sh:]uh<K∆h=hh)]ubhu)År‘��}r’��(h.Xµ���*Figure 4.* Sequence diagram for use case 06. Synchronize content between
Member and Coordinating Node. Detail for the *processObject()* step provided
in the activity diagram below.h/j†��h0h1h2hyh4}r÷��(h8]h9]h7]h6]h:]uh<K«h=hh)]r◊��(cdocutils.nodes
emphasis
rÿ��)ÅrŸ��}r⁄��(h.X���*Figure 4.*h4}r€��(h8]h9]h7]h6]h:]uh/j‘��h)]r‹��hQX	���Figure 4.r›��ÖÅrfi��}rfl��(h.U�h/jŸ��ubah2Uemphasisr‡��ubhQXl��� Sequence diagram for use case 06. Synchronize content between
Member and Coordinating Node. Detail for the r·��ÖÅr‚��}r„��(h.Xl��� Sequence diagram for use case 06. Synchronize content between
Member and Coordinating Node. Detail for the h/j‘��ubjÿ��)År‰��}rÂ��(h.X���*processObject()*h4}rÊ��(h8]h9]h7]h6]h:]uh/j‘��h)]rÁ��hQX���processObject()rË��ÖÅrÈ��}rÍ��(h.U�h/j‰��ubah2j‡��ubhQX-��� step provided
in the activity diagram below.rÎ��ÖÅrÏ��}rÌ��(h.X-��� step provided
in the activity diagram below.h/j‘��ubeubjÃ��)ÅrÓ��}rÔ��(h.X���.. image:: images/06_act.png
h/j†��h0h1h2jœ��h4}r��(UuriX!���design/UseCases/images/06_act.pngrÒ��h6]h7]h8]h9]j“��}rÚ��U*jÒ��sh:]uh<KÃh=hh)]ubhu)ÅrÛ��}rÙ��(h.Xá���*Figure 5.* Activity diagram indicating the execution flow after attempting to
retrieve the system metadata for the object from the CN.h/j†��h0h1h2hyh4}rı��(h8]h9]h7]h6]h:]uh<KÕh=hh)]rˆ��(jÿ��)År˜��}r¯��(h.X���*Figure 5.*h4}r˘��(h8]h9]h7]h6]h:]uh/jÛ��h)]r˙��hQX	���Figure 5.r˚��ÖÅr¸��}r˝��(h.U�h/j˜��ubah2j‡��ubhQX|��� Activity diagram indicating the execution flow after attempting to
retrieve the system metadata for the object from the CN.r˛��ÖÅrˇ��}r���(h.X|��� Activity diagram indicating the execution flow after attempting to
retrieve the system metadata for the object from the CN.h/j��ubeubcdocutils.nodes
comment
r��)År��}r��(h.XÓ��@startuml images/06_seq.png
skinparam notebordercolor #AAAAAA
skinparam notefontcolor #222222

participant "Repl Task Queue" as cn_repl << Cluster >>
participant "Replication" as cn_replication << CN >>
participant "Index Task Queue" as cn_index << Cluster >>
participant "Indexer" as cn_indexer << CN >>
participant "Object Store" as cn_objs << CN >>
participant "System Metadata Map" as cn_sysmeta << Cluster >>
participant "Sync Task Queue" as cn_queue << CN >>
participant "Node Map" as cn_nodes << Cluster >>
participant "Synchronization" as cn_sync << CN >>
participant "Read API" as mn_read << MN >>

'm_crud -> c_notify: notify(session, PID, OBJECT_CREATED)
'c_notify -> cn_queue: addTask(SyncTask, node, PID)
'note right
'  notification triggered by successful
'  create operation on MN.
'end note

'm_rep -> c_notify: setReplicationStatus(token, PID, COMPLETE)
'c_notify -> cn_queue: addTask(SyncTask, node, PID)
'note right
'  notification triggered by completed
'  replication operation on MN.
'end note

group populateSynchronizationQueue
  cn_sync -> cn_nodes: lock(node_id)
  activate cn_sync #D74F57
  note right
    Start of synchronization
    process triggered by quartz
  end note
  activate cn_nodes #D74F57
  cn_nodes --> cn_sync: OK
  cn_sync -> cn_nodes: getLastUpdateTime(node_id)
  activate cn_nodes #D74F57
  note right
    The Node Map is a hash of
    (node_id, Node), use Hazelcast query
    Map.values() passing in SqlPredicate
  end note
  cn_nodes --> cn_sync: startTime
  deactivate cn_nodes

  cn_sync -> mn_read: listObjects(session, startTime, ...)
  activate mn_read #D74F57
  cn_sync <-- mn_read: ObjectList
  deactivate mn_read
  loop for each PID
    cn_sync -> cn_sync: createTask(PID)
  note right
    Each SyncTask implements Callable
    and will be submitted to the
    ExecutorService to be executed
    on a CN
  end note

    cn_sync -> cn_queue: offer(taskid, SyncTask)
    activate cn_queue #D74F57
    cn_queue --> cn_sync: OK
    deactivate cn_queue
  end
  note right
    adding SyncTasks should fail if PID is already
    in the list and the PID is NOT locked. This enables
    very recent updates to a PID to occur and be
    correctly managed by the overall synchronization
    process.
  end note
  cn_sync -> cn_nodes: setLastUpdateTime(node_id, startTime)
  activate cn_nodes #D74F57
  cn_nodes --> cn_sync: OK
  deactivate cn_nodes

  cn_sync -> cn_nodes: unlock(node_id)
  deactivate cn_sync
  deactivate cn_nodes
end

group processSynchronizationQueue
  cn_sync -> cn_sync: entryAdded(EntryEvent<PID, SyncTask>)
  note left
    Synchronization implements EntryListener,
    monitors the Sync Queue for changes.
  end note
  activate cn_sync #D74F57
  cn_sync -> cn_queue: poll(timeout)
  activate cn_queue #D74F57
  note right
    Only one CN will win the poll and
    process the SyncTask
  end note
  cn_queue --> cn_sync: SyncTask
  deactivate cn_queue
  cn_sync -> cn_sync: ExecutorService.submit(SyncTask)
  cn_sync -> cn_sysmeta: lock(PID)
  activate cn_sysmeta #D74F57
  group ProcessPID( PID )
    cn_sync -> mn_read: getSystemMetadata(PID)
    mn_read -> cn_sync: SystemMetadata
    cn_sync -> mn_read: get( PID )
    mn_read -> cn_sync: object
    cn_sync -> cn_sync: work
    note right
      Check for new object,
      updates to properties
    end note
  end
  cn_sync -> cn_objs: createOrUpdate(session, PID, object, SystemMetadata)
  activate cn_objs #D74F57
  cn_indexer -> cn_indexer: entryAdded()
  activate cn_indexer #D74F57
  note left
    Indexer implements EntryListener,
    monitors the System Metadata Map
    for inserts, updates, deletes.
  end note
  cn_indexer -> cn_indexer: createTask(PID)
  note left
    Each IndexTask implements Callable
    and will be submitted to the
    ExecutorService to be executed
    on the local CN
  end note
  cn_indexer -> cn_index: offer(taskid, IndexTask)
  activate cn_index #D74F57
  cn_objs --> cn_sync: OK
  deactivate cn_objs
  cn_sync -> cn_sysmeta: unlock(PID)
  deactivate cn_queue
  deactivate cn_sysmeta
  cn_replication -> cn_replication: entryAdded()
  activate cn_replication #D74F57
  note left
    Indexer implements EntryListener,
    monitors the System Metadata Map
    for inserts, updates, deletes.
  end note
  cn_replication -> cn_replication: createTask(PID)
  note left
    Each ReplTask implements Callable
    and will be submitted to the
    ExecutorService to be executed
    on the a CN
  end note
  cn_replication -> cn_repl: offer(taskid, ReplTask)
  activate cn_repl #D74F57
  deactivate cn_queue
end
deactivate cn_sync
@endumlh/j†��h0h1h2Ucommentr��h4}r��(U	xml:spacer��Upreserver��h6]h7]h8]h9]h:]uh<Mnh=hh)]r��hQXÓ��@startuml images/06_seq.png
skinparam notebordercolor #AAAAAA
skinparam notefontcolor #222222

participant "Repl Task Queue" as cn_repl << Cluster >>
participant "Replication" as cn_replication << CN >>
participant "Index Task Queue" as cn_index << Cluster >>
participant "Indexer" as cn_indexer << CN >>
participant "Object Store" as cn_objs << CN >>
participant "System Metadata Map" as cn_sysmeta << Cluster >>
participant "Sync Task Queue" as cn_queue << CN >>
participant "Node Map" as cn_nodes << Cluster >>
participant "Synchronization" as cn_sync << CN >>
participant "Read API" as mn_read << MN >>

'm_crud -> c_notify: notify(session, PID, OBJECT_CREATED)
'c_notify -> cn_queue: addTask(SyncTask, node, PID)
'note right
'  notification triggered by successful
'  create operation on MN.
'end note

'm_rep -> c_notify: setReplicationStatus(token, PID, COMPLETE)
'c_notify -> cn_queue: addTask(SyncTask, node, PID)
'note right
'  notification triggered by completed
'  replication operation on MN.
'end note

group populateSynchronizationQueue
  cn_sync -> cn_nodes: lock(node_id)
  activate cn_sync #D74F57
  note right
    Start of synchronization
    process triggered by quartz
  end note
  activate cn_nodes #D74F57
  cn_nodes --> cn_sync: OK
  cn_sync -> cn_nodes: getLastUpdateTime(node_id)
  activate cn_nodes #D74F57
  note right
    The Node Map is a hash of
    (node_id, Node), use Hazelcast query
    Map.values() passing in SqlPredicate
  end note
  cn_nodes --> cn_sync: startTime
  deactivate cn_nodes

  cn_sync -> mn_read: listObjects(session, startTime, ...)
  activate mn_read #D74F57
  cn_sync <-- mn_read: ObjectList
  deactivate mn_read
  loop for each PID
    cn_sync -> cn_sync: createTask(PID)
  note right
    Each SyncTask implements Callable
    and will be submitted to the
    ExecutorService to be executed
    on a CN
  end note

    cn_sync -> cn_queue: offer(taskid, SyncTask)
    activate cn_queue #D74F57
    cn_queue --> cn_sync: OK
    deactivate cn_queue
  end
  note right
    adding SyncTasks should fail if PID is already
    in the list and the PID is NOT locked. This enables
    very recent updates to a PID to occur and be
    correctly managed by the overall synchronization
    process.
  end note
  cn_sync -> cn_nodes: setLastUpdateTime(node_id, startTime)
  activate cn_nodes #D74F57
  cn_nodes --> cn_sync: OK
  deactivate cn_nodes

  cn_sync -> cn_nodes: unlock(node_id)
  deactivate cn_sync
  deactivate cn_nodes
end

group processSynchronizationQueue
  cn_sync -> cn_sync: entryAdded(EntryEvent<PID, SyncTask>)
  note left
    Synchronization implements EntryListener,
    monitors the Sync Queue for changes.
  end note
  activate cn_sync #D74F57
  cn_sync -> cn_queue: poll(timeout)
  activate cn_queue #D74F57
  note right
    Only one CN will win the poll and
    process the SyncTask
  end note
  cn_queue --> cn_sync: SyncTask
  deactivate cn_queue
  cn_sync -> cn_sync: ExecutorService.submit(SyncTask)
  cn_sync -> cn_sysmeta: lock(PID)
  activate cn_sysmeta #D74F57
  group ProcessPID( PID )
    cn_sync -> mn_read: getSystemMetadata(PID)
    mn_read -> cn_sync: SystemMetadata
    cn_sync -> mn_read: get( PID )
    mn_read -> cn_sync: object
    cn_sync -> cn_sync: work
    note right
      Check for new object,
      updates to properties
    end note
  end
  cn_sync -> cn_objs: createOrUpdate(session, PID, object, SystemMetadata)
  activate cn_objs #D74F57
  cn_indexer -> cn_indexer: entryAdded()
  activate cn_indexer #D74F57
  note left
    Indexer implements EntryListener,
    monitors the System Metadata Map
    for inserts, updates, deletes.
  end note
  cn_indexer -> cn_indexer: createTask(PID)
  note left
    Each IndexTask implements Callable
    and will be submitted to the
    ExecutorService to be executed
    on the local CN
  end note
  cn_indexer -> cn_index: offer(taskid, IndexTask)
  activate cn_index #D74F57
  cn_objs --> cn_sync: OK
  deactivate cn_objs
  cn_sync -> cn_sysmeta: unlock(PID)
  deactivate cn_queue
  deactivate cn_sysmeta
  cn_replication -> cn_replication: entryAdded()
  activate cn_replication #D74F57
  note left
    Indexer implements EntryListener,
    monitors the System Metadata Map
    for inserts, updates, deletes.
  end note
  cn_replication -> cn_replication: createTask(PID)
  note left
    Each ReplTask implements Callable
    and will be submitted to the
    ExecutorService to be executed
    on the a CN
  end note
  cn_replication -> cn_repl: offer(taskid, ReplTask)
  activate cn_repl #D74F57
  deactivate cn_queue
end
deactivate cn_sync
@endumlr	��ÖÅr
��}r��(h.U�h/j��ubaubj��)År��}r
��(h.X��@startuml images/06_act.png
(*) --> [processObject()] if "CNRead.getSystemMetadata(PID)" then
  if "Duplicate Object?" then
    -->[Yes] "Duplicate Content"
    --> "Update Replica Info"
    --> "Store System Metadata"
    --> "Notify watchers\nIndex, Replication"
    --> (*)
  else
    -->[No] "Error: Duplicate Identifier"
    --> "Notify MN"
    --> (*)
  endif

else
  ->[FAIL] "New content"
  note right
    The content is unknown to DataONE
    so needs to be examined, its
    location recorded, and optionally
    retrieved and stored on the CN.
  end note
  if "Science Metadata \n or Resource Map?" then
    ->[yes] "Get object from MN"
    --> "Store object on CN"
    --> "Update Replica Info"
  else
    -->[no] "Update Replica Info"
  endif
endif
@endumlh/j†��h0h1h2j��h4}r��(j��j��h6]h7]h8]h9]h:]uh<Mêh=hh)]r��hQX��@startuml images/06_act.png
(*) --> [processObject()] if "CNRead.getSystemMetadata(PID)" then
  if "Duplicate Object?" then
    -->[Yes] "Duplicate Content"
    --> "Update Replica Info"
    --> "Store System Metadata"
    --> "Notify watchers\nIndex, Replication"
    --> (*)
  else
    -->[No] "Error: Duplicate Identifier"
    --> "Notify MN"
    --> (*)
  endif

else
  ->[FAIL] "New content"
  note right
    The content is unknown to DataONE
    so needs to be examined, its
    location recorded, and optionally
    retrieved and stored on the CN.
  end note
  if "Science Metadata \n or Resource Map?" then
    ->[yes] "Get object from MN"
    --> "Store object on CN"
    --> "Update Replica Info"
  else
    -->[no] "Update Replica Info"
  endif
endif
@endumlr��ÖÅr��}r��(h.U�h/j��ubaubeubh>)År��}r��(h.U�h/h?h0h1h2hCh4}r��(h8]h9]h7]h6]r��hah:]r��hauh<Míh=hh)]r��(hJ)År��}r��(h.X���Implementationr��h/j��h0h1h2hNh4}r��(h8]h9]h7]h6]h:]uh<Míh=hh)]r��hQX���Implementationr��ÖÅr��}r ��(h.j��h/j��ubaubhu)År!��}r"��(h.X·���The Member Node synchronization process will operate in an asynchronous
manner, with a task queue on the CN service containing a list of objects that
need to be added to the coordinating node populated through two mechanisms:r#��h/j��h0h1h2hyh4}r$��(h8]h9]h7]h6]h:]uh<Mîh=hh)]r%��hQX·���The Member Node synchronization process will operate in an asynchronous
manner, with a task queue on the CN service containing a list of objects that
need to be added to the coordinating node populated through two mechanisms:r&��ÖÅr'��}r(��(h.j#��h/j!��ubaubcdocutils.nodes
enumerated_list
r)��)År*��}r+��(h.U�h/j��h0h1h2Uenumerated_listr,��h4}r-��(Usuffixr.��U.h6]h7]h8]Uprefixr/��U�h9]h:]Uenumtyper0��Uarabicr1��uh<Mòh=hh)]r2��(h¬)År3��}r4��(h.X.���scheduled calls to :func:`MNRead.listObjects`
h/j*��h0h1h2h∆h4}r5��(h8]h9]h7]h6]h:]uh<Nh=hh)]r6��hu)År7��}r8��(h.X-���scheduled calls to :func:`MNRead.listObjects`h/j3��h0h1h2hyh4}r9��(h8]h9]h7]h6]h:]uh<Mòh)]r:��(hQX���scheduled calls to r;��ÖÅr<��}r=��(h.X���scheduled calls to h/j7��ubcsphinx.addnodes
pending_xref
r>��)År?��}r@��(h.X���:func:`MNRead.listObjects`rA��h/j7��h0h1h2Upending_xrefrB��h4}rC��(UreftypeX���funcUrefwarnrD��âU	reftargetrE��X���MNRead.listObjectsU	refdomainX���pyrF��h6]h7]Urefexplicitâh8]h9]h:]UrefdocrG��X���design/UseCases/06_ucrH��Upy:classrI��NU	py:modulerJ��Nuh<Mòh)]rK��cdocutils.nodes
literal
rL��)ÅrM��}rN��(h.jA��h4}rO��(h8]h9]rP��(UxrefrQ��jF��X���py-funcrR��eh7]h6]h:]uh/j?��h)]rS��hQX���MNRead.listObjects()rT��ÖÅrU��}rV��(h.U�h/jM��ubah2UliteralrW��ubaubeubaubh¬)ÅrX��}rY��(h.Xd���through a new notification API that is called by a Member Node after a
successful create operation.
h/j*��h0h1h2h∆h4}rZ��(h8]h9]h7]h6]h:]uh<Nh=hh)]r[��hu)År\��}r]��(h.Xc���through a new notification API that is called by a Member Node after a
successful create operation.r^��h/jX��h0h1h2hyh4}r_��(h8]h9]h7]h6]h:]uh<Möh)]r`��hQXc���through a new notification API that is called by a Member Node after a
successful create operation.ra��ÖÅrb��}rc��(h.j^��h/j\��ubaubaubeubhu)Ård��}re��(h.XÌ���The task queue is processed by a pool of workers which retrieve the content
using :func:`MNRead.get` and :func:`MNRead.getSystemMetadata`, and store the
content in the Coordinating Node using the internal :func:`CNStorage.create`
method.h/j��h0h1h2hyh4}rf��(h8]h9]h7]h6]h:]uh<Mùh=hh)]rg��(hQXR���The task queue is processed by a pool of workers which retrieve the content
using rh��ÖÅri��}rj��(h.XR���The task queue is processed by a pool of workers which retrieve the content
using h/jd��ubj>��)Årk��}rl��(h.X���:func:`MNRead.get`rm��h/jd��h0h1h2jB��h4}rn��(UreftypeX���funcjD��âjE��X
���MNRead.getU	refdomainX���pyro��h6]h7]Urefexplicitâh8]h9]h:]jG��jH��jI��NjJ��Nuh<Mùh)]rp��jL��)Årq��}rr��(h.jm��h4}rs��(h8]h9]rt��(jQ��jo��X���py-funcru��eh7]h6]h:]uh/jk��h)]rv��hQX���MNRead.get()rw��ÖÅrx��}ry��(h.U�h/jq��ubah2jW��ubaubhQX��� and rz��ÖÅr{��}r|��(h.X��� and h/jd��ubj>��)År}��}r~��(h.X ���:func:`MNRead.getSystemMetadata`r��h/jd��h0h1h2jB��h4}rÄ��(UreftypeX���funcjD��âjE��X���MNRead.getSystemMetadataU	refdomainX���pyrÅ��h6]h7]Urefexplicitâh8]h9]h:]jG��jH��jI��NjJ��Nuh<Mùh)]rÇ��jL��)ÅrÉ��}rÑ��(h.j��h4}rÖ��(h8]h9]rÜ��(jQ��jÅ��X���py-funcrá��eh7]h6]h:]uh/j}��h)]rà��hQX���MNRead.getSystemMetadata()râ��ÖÅrä��}rã��(h.U�h/jÉ��ubah2jW��ubaubhQXD���, and store the
content in the Coordinating Node using the internal rå��ÖÅrç��}ré��(h.XD���, and store the
content in the Coordinating Node using the internal h/jd��ubj>��)Årè��}rê��(h.X���:func:`CNStorage.create`rë��h/jd��h0h1h2jB��h4}rí��(UreftypeX���funcjD��âjE��X���CNStorage.createU	refdomainX���pyrì��h6]h7]Urefexplicitâh8]h9]h:]jG��jH��jI��NjJ��Nuh<Mùh)]rî��jL��)Årï��}rñ��(h.jë��h4}ró��(h8]h9]rò��(jQ��jì��X���py-funcrô��eh7]h6]h:]uh/jè��h)]rö��hQX���CNStorage.create()rõ��ÖÅrú��}rù��(h.U�h/jï��ubah2jW��ubaubhQX���
method.rû��ÖÅrü��}r†��(h.X���
method.h/jd��ubeubhu)År°��}r¢��(h.X…���The initial implementation will focus on using only scheduled calls to
:func:`MNRead.listObjects` to populate the task queue. The scheduling
information shall be set during the MN registration process.h/j��h0h1h2hyh4}r£��(h8]h9]h7]h6]h:]uh<M¢h=hh)]r§��(hQXG���The initial implementation will focus on using only scheduled calls to
r•��ÖÅr¶��}rß��(h.XG���The initial implementation will focus on using only scheduled calls to
h/j°��ubj>��)År®��}r©��(h.X���:func:`MNRead.listObjects`r™��h/j°��h0h1h2jB��h4}r´��(UreftypeX���funcjD��âjE��X���MNRead.listObjectsU	refdomainX���pyr¨��h6]h7]Urefexplicitâh8]h9]h:]jG��jH��jI��NjJ��Nuh<M¢h)]r≠��jL��)ÅrÆ��}rØ��(h.j™��h4}r∞��(h8]h9]r±��(jQ��j¨��X���py-funcr≤��eh7]h6]h:]uh/j®��h)]r≥��hQX���MNRead.listObjects()r¥��ÖÅrµ��}r∂��(h.U�h/jÆ��ubah2jW��ubaubhQXh��� to populate the task queue. The scheduling
information shall be set during the MN registration process.r∑��ÖÅr∏��}rπ��(h.Xh��� to populate the task queue. The scheduling
information shall be set during the MN registration process.h/j°��ubeubhu)År∫��}rª��(h.X'��Multiple threads should be able to process the task queue, and the number of
threads should be adjustable by an administrator. The total number of threads
hitting on a particular MN should be limited, the limit should be specified in
the MN capabilities recorded during the registration process.rº��h/j��h0h1h2hyh4}rΩ��(h8]h9]h7]h6]h:]uh<M¶h=hh)]ræ��hQX'��Multiple threads should be able to process the task queue, and the number of
threads should be adjustable by an administrator. The total number of threads
hitting on a particular MN should be limited, the limit should be specified in
the MN capabilities recorded during the registration process.rø��ÖÅr¿��}r¡��(h.jº��h/j∫��ubaubhu)År¬��}r√��(h.X��The tasks queue will need to be thread safe. In the future it may be feasible
to use a distributed queue (based on Apache MQ or Zookeeper for example;
shared across the CNs) so that processes on any CN can issue the
:func:`MN_get` and :func:`MN_getSystemMetadata` requests.h/j��h0h1h2hyh4}rƒ��(h8]h9]h7]h6]h:]uh<M´h=hh)]r≈��(hQXÿ���The tasks queue will need to be thread safe. In the future it may be feasible
to use a distributed queue (based on Apache MQ or Zookeeper for example;
shared across the CNs) so that processes on any CN can issue the
r∆��ÖÅr«��}r»��(h.Xÿ���The tasks queue will need to be thread safe. In the future it may be feasible
to use a distributed queue (based on Apache MQ or Zookeeper for example;
shared across the CNs) so that processes on any CN can issue the
h/j¬��ubj>��)År…��}r ��(h.X���:func:`MN_get`rÀ��h/j¬��h0h1h2jB��h4}rÃ��(UreftypeX���funcjD��âjE��X���MN_getU	refdomainX���pyrÕ��h6]h7]Urefexplicitâh8]h9]h:]jG��jH��jI��NjJ��Nuh<M´h)]rŒ��jL��)Årœ��}r–��(h.jÀ��h4}r—��(h8]h9]r“��(jQ��jÕ��X���py-funcr”��eh7]h6]h:]uh/j…��h)]r‘��hQX���MN_get()r’��ÖÅr÷��}r◊��(h.U�h/jœ��ubah2jW��ubaubhQX��� and rÿ��ÖÅrŸ��}r⁄��(h.X��� and h/j¬��ubj>��)År€��}r‹��(h.X���:func:`MN_getSystemMetadata`r›��h/j¬��h0h1h2jB��h4}rfi��(UreftypeX���funcjD��âjE��X���MN_getSystemMetadataU	refdomainX���pyrfl��h6]h7]Urefexplicitâh8]h9]h:]jG��jH��jI��NjJ��Nuh<M´h)]r‡��jL��)År·��}r‚��(h.j›��h4}r„��(h8]h9]r‰��(jQ��jfl��X���py-funcrÂ��eh7]h6]h:]uh/j€��h)]rÊ��hQX���MN_getSystemMetadata()rÁ��ÖÅrË��}rÈ��(h.U�h/j·��ubah2jW��ubaubhQX
��� requests.rÍ��ÖÅrÎ��}rÏ��(h.X
��� requests.h/j¬��ubeubhu)ÅrÌ��}rÓ��(h.X‘���There will likely be a significant load on the system for indexing as new
content is being added - this is the case for both Mercury and Metacat.
Options for disabling indexing for bulk load should be considered.rÔ��h/j��h0h1h2hyh4}r��(h8]h9]h7]h6]h:]uh<M∞h=hh)]rÒ��hQX‘���There will likely be a significant load on the system for indexing as new
content is being added - this is the case for both Mercury and Metacat.
Options for disabling indexing for bulk load should be considered.rÚ��ÖÅrÛ��}rÙ��(h.jÔ��h/jÌ��ubaubhu)Årı��}rˆ��(h.Xú���The synchronization process will certainly be more sophisticated down the road,
but for now (i.e. prototype implementation) simplicity should be a priority:r˜��h/j��h0h1h2hyh4}r¯��(h8]h9]h7]h6]h:]uh<M¥h=hh)]r˘��hQXú���The synchronization process will certainly be more sophisticated down the road,
but for now (i.e. prototype implementation) simplicity should be a priority:r˙��ÖÅr˚��}r¸��(h.j˜��h/jı��ubaubhª)År˝��}r˛��(h.U�h/j��h0h1h2hæh4}rˇ��(h¿X���-h6]h7]h8]h9]h:]uh<M∑h=hh)]r���(h¬)År��}r��(h.Xd���MNs are only scanned by the CN where they are registered (restriction to be
dropped down the road).
h/j˝��h0h1h2h∆h4}r��(h8]h9]h7]h6]h:]uh<Nh=hh)]r��hu)År��}r��(h.Xc���MNs are only scanned by the CN where they are registered (restriction to be
dropped down the road).r��h/j��h0h1h2hyh4}r��(h8]h9]h7]h6]h:]uh<M∑h)]r	��hQXc���MNs are only scanned by the CN where they are registered (restriction to be
dropped down the road).r
��ÖÅr��}r��(h.j��h/j��ubaubaubh¬)År
��}r��(h.Xe���listObjects is polled (with max objects set to some limit), with results
stored in a queue on the CN
h/j˝��h0h1h2h∆h4}r��(h8]h9]h7]h6]h:]uh<Nh=hh)]r��hu)År��}r��(h.Xd���listObjects is polled (with max objects set to some limit), with results
stored in a queue on the CNr��h/j
��h0h1h2hyh4}r��(h8]h9]h7]h6]h:]uh<M∫h)]r��hQXd���listObjects is polled (with max objects set to some limit), with results
stored in a queue on the CNr��ÖÅr��}r��(h.j��h/j��ubaubaubh¬)År��}r��(h.X]���as the queue is processed and shrinks to some size, then listObjects process
is polled again
h/j˝��h0h1h2h∆h4}r��(h8]h9]h7]h6]h:]uh<Nh=hh)]r��hu)År��}r��(h.X\���as the queue is processed and shrinks to some size, then listObjects process
is polled againr��h/j��h0h1h2hyh4}r ��(h8]h9]h7]h6]h:]uh<MΩh)]r!��hQX\���as the queue is processed and shrinks to some size, then listObjects process
is polled againr"��ÖÅr#��}r$��(h.j��h/j��ubaubaubh¬)År%��}r&��(h.X•���the queue is processed by n worker threads, with each task being to pop an
entry from the queue, pull the content from the target, and store it in the
object store.
h/j˝��h0h1h2h∆h4}r'��(h8]h9]h7]h6]h:]uh<Nh=hh)]r(��hu)År)��}r*��(h.X§���the queue is processed by n worker threads, with each task being to pop an
entry from the queue, pull the content from the target, and store it in the
object store.r+��h/j%��h0h1h2hyh4}r,��(h8]h9]h7]h6]h:]uh<M¿h)]r-��hQX§���the queue is processed by n worker threads, with each task being to pop an
entry from the queue, pull the content from the target, and store it in the
object store.r.��ÖÅr/��}r0��(h.j+��h/j)��ubaubaubh¬)År1��}r2��(h.X��when there is some number of new entries in the object store, then the
mercury indexing process needs to kick in. Ideally this should be updating
rather than re-creating the index at each pass. This may be initiated by a
cron job for the prototype implementation.
h/j˝��h0h1h2h∆h4}r3��(h8]h9]h7]h6]h:]uh<Nh=hh)]r4��hu)År5��}r6��(h.X��when there is some number of new entries in the object store, then the
mercury indexing process needs to kick in. Ideally this should be updating
rather than re-creating the index at each pass. This may be initiated by a
cron job for the prototype implementation.r7��h/j1��h0h1h2hyh4}r8��(h8]h9]h7]h6]h:]uh<Mƒh)]r9��hQX��when there is some number of new entries in the object store, then the
mercury indexing process needs to kick in. Ideally this should be updating
rather than re-creating the index at each pass. This may be initiated by a
cron job for the prototype implementation.r:��ÖÅr;��}r<��(h.j7��h/j5��ubaubaubeubhu)År=��}r>��(h.Xm��This fairly simplistic approach should be enough to get things started. Not
ideal, but should suffice to get some data moving around. To implement, there
is need for a few new components - a queue, a place to store state
information, the code that does the polling, the code that does the object
retrieval, the worker thread code, and an overall controller service.r?��h/j��h0h1h2hyh4}r@��(h8]h9]h7]h6]h:]uh<M…h=hh)]rA��hQXm��This fairly simplistic approach should be enough to get things started. Not
ideal, but should suffice to get some data moving around. To implement, there
is need for a few new components - a queue, a place to store state
information, the code that does the polling, the code that does the object
retrieval, the worker thread code, and an overall controller service.rB��ÖÅrC��}rD��(h.j?��h/j=��ubaubh>)ÅrE��}rF��(h.U�h/j��h0h1h2hCh4}rG��(h8]h9]h7]h6]rH��h ah:]rI��h
auh<M—h=hh)]rJ��(hJ)ÅrK��}rL��(h.X���In more detailrM��h/jE��h0h1h2hNh4}rN��(h8]h9]h7]h6]h:]uh<M—h=hh)]rO��hQX���In more detailrP��ÖÅrQ��}rR��(h.jM��h/jK��ubaubhu)ÅrS��}rT��(h.X¸���Retrieve objects from Member Nodes. getNextMemberNode() gets the next Member
Node to work on - this could be serial selection from the list of registered
MNs, could be based on the last update, perhaps from some hint returned in
:func:`MN_health.ping`.h/jE��h0h1h2hyh4}rU��(h8]h9]h7]h6]h:]uh<M”h=hh)]rV��(hQXÂ���Retrieve objects from Member Nodes. getNextMemberNode() gets the next Member
Node to work on - this could be serial selection from the list of registered
MNs, could be based on the last update, perhaps from some hint returned in
rW��ÖÅrX��}rY��(h.XÂ���Retrieve objects from Member Nodes. getNextMemberNode() gets the next Member
Node to work on - this could be serial selection from the list of registered
MNs, could be based on the last update, perhaps from some hint returned in
h/jS��ubj>��)ÅrZ��}r[��(h.X���:func:`MN_health.ping`r\��h/jS��h0h1h2jB��h4}r]��(UreftypeX���funcjD��âjE��X���MN_health.pingU	refdomainX���pyr^��h6]h7]Urefexplicitâh8]h9]h:]jG��jH��jI��NjJ��Nuh<M”h)]r_��jL��)År`��}ra��(h.j\��h4}rb��(h8]h9]rc��(jQ��j^��X���py-funcrd��eh7]h6]h:]uh/jZ��h)]re��hQX���MN_health.ping()rf��ÖÅrg��}rh��(h.U�h/j`��ubah2jW��ubaubhQX���.ÖÅri��}rj��(h.X���.h/jS��ubeubhu)Årk��}rl��(h.X���New structures::rm��h/jE��h0h1h2hyh4}rn��(h8]h9]h7]h6]h:]uh<Mÿh=hh)]ro��hQX���New structures:rp��ÖÅrq��}rr��(h.X���New structures:h/jk��ubaubcdocutils.nodes
literal_block
rs��)Årt��}ru��(h.X∆���MemberNodeState
  String url;  //URL of the member node
  DateTime lastCheckTime;  // time stamp for when the MN was last polled for listOjects


RetrievalTask
  String url;
  ObjectInfo objectInfo;h/jE��h0h1h2U
literal_blockrv��h4}rw��(j��j��h6]h7]h8]h9]h:]uh<M⁄h=hh)]rx��hQX∆���MemberNodeState
  String url;  //URL of the member node
  DateTime lastCheckTime;  // time stamp for when the MN was last polled for listOjects


RetrievalTask
  String url;
  ObjectInfo objectInfo;ry��ÖÅrz��}r{��(h.U�h/jt��ubaubhu)År|��}r}��(h.X/���Populating the harvest task queue pseudo code::r~��h/jE��h0h1h2hyh4}r��(h8]h9]h7]h6]h:]uh<MÂh=hh)]rÄ��hQX.���Populating the harvest task queue pseudo code:rÅ��ÖÅrÇ��}rÉ��(h.X.���Populating the harvest task queue pseudo code:h/j|��ubaubjs��)ÅrÑ��}rÖ��(h.Xê��void populateTaskQueue ()
{
  ThreadSafeQueue taskQueue = getSharedTaskQueue()
  while ( keepRunning() )
  {
    if (taskQueue.length < MIN_TASKS)
    {
      MemberNodeState = getNextMemberNode();

      // This could be executed in a different thread, which would
      // enable multiple MNs to be polled at a tme if necessary
      int numNewEntries = loadMNTasks(taskQueue, mnState);
    }
  }
}h/jE��h0h1h2jv��h4}rÜ��(j��j��h6]h7]h8]h9]h:]uh<MÁh=hh)]rá��hQXê��void populateTaskQueue ()
{
  ThreadSafeQueue taskQueue = getSharedTaskQueue()
  while ( keepRunning() )
  {
    if (taskQueue.length < MIN_TASKS)
    {
      MemberNodeState = getNextMemberNode();

      // This could be executed in a different thread, which would
      // enable multiple MNs to be polled at a tme if necessary
      int numNewEntries = loadMNTasks(taskQueue, mnState);
    }
  }
}rà��ÖÅrâ��}rä��(h.U�h/jÑ��ubaubhu)Årã��}rå��(h.X/���Processing the harvest task queue pseudo code::rç��h/jE��h0h1h2hyh4}ré��(h8]h9]h7]h6]h:]uh<M¯h=hh)]rè��hQX.���Processing the harvest task queue pseudo code:rê��ÖÅrë��}rí��(h.X.���Processing the harvest task queue pseudo code:h/jã��ubaubjs��)Årì��}rî��(h.X]��int loadMNTasks(ThreadSafeQueue taskQueue, MemberNodeState mnState)
{
  AuthToken token = getSystemToken();
  D1cient client = D1Client( mnState.url );

  // get a list of all the new entries since lastCheckTime
  DateTime tstamp = now();

  // Will actually need to loop here to page through results
  ObjectList objectList = client.listObjects(token, mnState.lastCheckTime);
  for (ObjectInfo objinfo : objectList)
  {
    RetrievalTask task = RetrievalTask(mnState.url, objinfo)
    taskQueue.push(task)
  }
  mnState.lastCheckTime = tstamp;
  setMemberNodeState(mnState);
  return objectList.length;
}h/jE��h0h1h2jv��h4}rï��(j��j��h6]h7]h8]h9]h:]uh<M˙h=hh)]rñ��hQX]��int loadMNTasks(ThreadSafeQueue taskQueue, MemberNodeState mnState)
{
  AuthToken token = getSystemToken();
  D1cient client = D1Client( mnState.url );

  // get a list of all the new entries since lastCheckTime
  DateTime tstamp = now();

  // Will actually need to loop here to page through results
  ObjectList objectList = client.listObjects(token, mnState.lastCheckTime);
  for (ObjectInfo objinfo : objectList)
  {
    RetrievalTask task = RetrievalTask(mnState.url, objinfo)
    taskQueue.push(task)
  }
  mnState.lastCheckTime = tstamp;
  setMemberNodeState(mnState);
  return objectList.length;
}ró��ÖÅrò��}rô��(h.U�h/jì��ubaubhu)Årö��}rõ��(h.X��Worker threads are responsible for retrieving each object listed in the
taskQueue. The number of workers should be adjustable. An obvious target for
improving efficiency is to enable requesting multiple objects in a single call
to a MN (later enhancement)::h/jE��h0h1h2hyh4}rú��(h8]h9]h7]h6]h:]uh<Mh=hh)]rù��hQX���Worker threads are responsible for retrieving each object listed in the
taskQueue. The number of workers should be adjustable. An obvious target for
improving efficiency is to enable requesting multiple objects in a single call
to a MN (later enhancement):rû��ÖÅrü��}r†��(h.X���Worker threads are responsible for retrieving each object listed in the
taskQueue. The number of workers should be adjustable. An obvious target for
improving efficiency is to enable requesting multiple objects in a single call
to a MN (later enhancement):h/jö��ubaubjs��)År°��}r¢��(h.XÂ��// object is the next entry from the taskQueue
bool retrieveObject(String mnurl, ObjectInfo object)
{
  Authtoken token = getSystemToken();
  D1Client client = D1Client(mnurl);
  SystemMetadata sysmeta = client.getSystemMetadata( object.identifier );
  if ( isScienceMetadata(sysmeta.objectClass) )
  {
    scimeta = client.get( object.identifier );
    CN_SERVICE.create( object.identifier, sysmeta, scimeta );
  }
  else
  {
    CN_SERVICE.create( object.identifier, sysmeta );
  }
}h/jE��h0h1h2jv��h4}r£��(j��j��h6]h7]h8]h9]h:]uh<Mh=hh)]r§��hQXÂ��// object is the next entry from the taskQueue
bool retrieveObject(String mnurl, ObjectInfo object)
{
  Authtoken token = getSystemToken();
  D1Client client = D1Client(mnurl);
  SystemMetadata sysmeta = client.getSystemMetadata( object.identifier );
  if ( isScienceMetadata(sysmeta.objectClass) )
  {
    scimeta = client.get( object.identifier );
    CN_SERVICE.create( object.identifier, sysmeta, scimeta );
  }
  else
  {
    CN_SERVICE.create( object.identifier, sysmeta );
  }
}r•��ÖÅr¶��}rß��(h.U�h/j°��ubaubh+)År®��}r©��(h.X°���.. _history: https://redmine.dataone.org/projects/d1/repository/changes/documents/Projects/cicore/architecture/api-documentation/source/design/UseCases/06_uc.txth/jE��h0h1h2h3h4}r™��(Urefurir´��Xî���https://redmine.dataone.org/projects/d1/repository/changes/documents/Projects/cicore/architecture/api-documentation/source/design/UseCases/06_uc.txth6]r¨��h(ah7]h8]h9]h:]r≠��hauh<M'h=hh)]ubeubeubeubeh.U�UtransformerrÆ��NU
footnote_refsrØ��}r∞��Urefnamesr±��}r≤��Usymbol_footnotesr≥��]r¥��Uautofootnote_refsrµ��]r∂��Usymbol_footnote_refsr∑��]r∏��U	citationsrπ��]r∫��h=hUcurrent_linerª��NUtransform_messagesrº��]rΩ��(cdocutils.nodes
system_message
ræ��)Årø��}r¿��(h.U�h4}r¡��(h8]UlevelKh6]h7]Usourceh1h9]h:]UlineKUtypeUINFOr¬��uh)]r√��hu)Årƒ��}r≈��(h.U�h4}r∆��(h8]h9]h7]h6]h:]uh/jø��h)]r«��hQX*���Hyperlink target "uc06" is not referenced.r»��ÖÅr…��}r ��(h.U�h/jƒ��ubah2hyubah2Usystem_messagerÀ��ubjæ��)ÅrÃ��}rÕ��(h.U�h4}rŒ��(h8]UlevelKh6]h7]Usourceh1h9]h:]UlineKUtypej¬��uh)]rœ��hu)År–��}r—��(h.U�h4}r“��(h8]h9]h7]h6]h:]uh/jÃ��h)]r”��hQX-���Hyperlink target "index-0" is not referenced.r‘��ÖÅr’��}r÷��(h.U�h/j–��ubah2hyubah2jÀ��ubjæ��)År◊��}rÿ��(h.U�h4}rŸ��(h8]UlevelKh6]h7]Usourceh1h9]h:]Utypej¬��uh)]r⁄��hu)År€��}r‹��(h.U�h4}r›��(h8]h9]h7]h6]h:]uh/j◊��h)]rfi��hQX-���Hyperlink target "index-1" is not referenced.rfl��ÖÅr‡��}r·��(h.U�h/j€��ubah2hyubah2jÀ��ubjæ��)År‚��}r„��(h.U�h4}r‰��(h8]UlevelKh6]h7]Usourceh1h9]h:]UlineM'Utypej¬��uh)]rÂ��hu)ÅrÊ��}rÁ��(h.U�h4}rË��(h8]h9]h7]h6]h:]uh/j‚��h)]rÈ��hQX-���Hyperlink target "history" is not referenced.rÍ��ÖÅrÎ��}rÏ��(h.U�h/jÊ��ubah2hyubah2jÀ��ubeUreporterrÌ��NUid_startrÓ��KU
autofootnotesrÔ��]r��U
citation_refsrÒ��}rÚ��Uindirect_targetsrÛ��]rÙ��Usettingsrı��(cdocutils.frontend
Values
rˆ��or˜��}r¯��(Ufootnote_backlinksr˘��KUrecord_dependenciesr˙��NUrfc_base_urlr˚��Uhttps://tools.ietf.org/html/r¸��U	tracebackr˝��àUpep_referencesr˛��NUstrip_commentsrˇ��NU
toc_backlinksr���Uentryr��U
language_coder��Uenr��U	datestampr��NUreport_levelr��KU_destinationr��NU
halt_levelr��KU
strip_classesr��NhNNUerror_encoding_error_handlerr	��Ubackslashreplacer
��Udebugr��NUembed_stylesheetr��âUoutput_encoding_error_handlerr
��Ustrictr��U
sectnum_xformr��KUdump_transformsr��NU
docinfo_xformr��KUwarning_streamr��NUpep_file_url_templater��Upep-%04dr��Uexit_status_levelr��KUconfigr��NUstrict_visitorr��NUcloak_email_addressesr��àUtrim_footnote_reference_spacer��âUenvr��NUdump_pseudo_xmlr��NUexpose_internalsr��NUsectsubtitle_xformr��âUsource_linkr��NUrfc_referencesr��NUoutput_encodingr ��Uutf-8r!��U
source_urlr"��NUinput_encodingr#��U	utf-8-sigr$��U_disable_configr%��NU	id_prefixr&��U�U	tab_widthr'��KUerror_encodingr(��UUTF-8r)��U_sourcer*��h1Ugettext_compactr+��àU	generatorr,��NUdump_internalsr-��NUsmart_quotesr.��âUpep_base_urlr/��U https://www.python.org/dev/peps/r0��Usyntax_highlightr1��Ulongr2��Uinput_encoding_error_handlerr3��j��Uauto_id_prefixr4��Uidr5��Udoctitle_xformr6��âUstrip_elements_with_classesr7��NU
_config_filesr8��]Ufile_insertion_enabledr9��àUraw_enabledr:��KU
dump_settingsr;��NubUsymbol_footnote_startr<��K�Uidsr=��}r>��(h%jF��hhfhj��hj��h#j†��h jE��h!hhh?h"jp��h&h≠j±��j≥��h]hfh$hÚh'h?h(j®��uUsubstitution_namesr?��}r@��h2h=h4}rA��(h8]h6]h7]Usourceh1h9]h:]uU	footnotesrB��]rC��UrefidsrD��}rE��(h]]rF��hcaj±��]rG��jÆ��ah']rH��h,auub.