`[docutils.nodesdocument)}( rawsourcechildren](hraw)}(h\newpageh]hText\newpage}(hhparenth uba attributes}(ids]classes]names]dupnames]backrefs]formatlatex xml:spacepreserveutagnameh source]/var/lib/jenkins/jobs/metacat_beta/workspace/metacat/docs/user/metacat/source/query-index.rstlineKhhhhubhsection)}(hhh](htitle)}(hMetacat Indexingh]hMetacat Indexing}(hh3hh1hhh'Nh)Nubah}(h]h]h]h]h ]uh&h/hh,hhh'h(h)Kubh paragraph)}(hMetacat v2.1 introduces support for building a SOLR index of Metacat content. While we continue to support the "pathquery" search mechanism, this will be phased out in favor of the more efficient SOLR query interface.h]hMetacat v2.1 introduces support for building a SOLR index of Metacat content. While we continue to support the “pathquery” search mechanism, this will be phased out in favor of the more efficient SOLR query interface.}(hhChhAhhh'Nh)Nubah}(h]h]h]h]h ]uh&h?h'h(h)Khh,hhubh@)}(h]Metacat deployments that opt to use the Metacat SOLR index will be able to take advantage of:h]h]Metacat deployments that opt to use the Metacat SOLR index will be able to take advantage of:}(hhQhhOhhh'Nh)Nubah}(h]h]h]h]h ]uh&h?h'h(h)K hh,hhubh bullet_list)}(hhh](h list_item)}(hfast search performanceh]h@)}(hhfh]hfast search performance}(hhfhhhubah}(h]h]h]h]h ]uh&h?h'h(h)Khhdubah}(h]h]h]h]h ]uh&hbhh_hhh'h(h)Nubhc)}(hbuilt-in paging featuresh]h@)}(hh}h]hbuilt-in paging features}(hh}hhubah}(h]h]h]h]h ]uh&h?h'h(h)Khh{ubah}(h]h]h]h]h ]uh&hbhh_hhh'h(h)Nubhc)}(h2customizable return formats (for advanced admins) h]h@)}(h1customizable return formats (for advanced admins)h]h1customizable return formats (for advanced admins)}(hhhhubah}(h]h]h]h]h ]uh&h?h'h(h)Khhubah}(h]h]h]h]h ]uh&hbhh_hhh'h(h)Nubeh}(h]h]h]h]h ]bullet*uh&h]h'h(h)Khh,hhubh+)}(hhh](h0)}(hIndexed documents and fieldsh]hIndexed documents and fields}(hhhhhhh'Nh)Nubah}(h]h]h]h]h ]uh&h/hhhhh'h(h)Kubh@)}(hqMetacat integrates the existing DataONE index library which includes many common metadata formats out-of-the-box:h]hqMetacat integrates the existing DataONE index library which includes many common metadata formats out-of-the-box:}(hhhhhhh'Nh)Nubah}(h]h]h]h]h ]uh&h?h'h(h)Khhhhubhenumerated_list)}(hhh](hc)}(hEMLh]h@)}(hhh]hEML}(hhhhubah}(h]h]h]h]h ]uh&h?h'h(h)Khhubah}(h]h]h]h]h ]uh&hbhhhhh'h(h)Nubhc)}(hFGDCh]h@)}(hhh]hFGDC}(hhhhubah}(h]h]h]h]h ]uh&h?h'h(h)Khhubah}(h]h]h]h]h ]uh&hbhhhhh'h(h)Nubhc)}(hDryad* h]h@)}(hDryad*h]hDryad*}(hj hjubah}(h]h]h]h]h ]uh&h?h'h(h)Khjubah}(h]h]h]h]h ]uh&hbhhhhh'h(h)Nubeh}(h]h]h]h]h ]enumtypearabicprefixhsuffix.uh&hhhhhh'h(h)Kubeh}(h]indexed-documents-and-fieldsah]h]indexed documents and fieldsah]h ]uh&h*hh,hhh'h(h)Kubh+)}(hhh](h0)}(hDefault indexed fieldsh]hDefault indexed fields}(hj4hj2hhh'Nh)Nubah}(h]h]h]h]h ]uh&h/hj/hhh'h(h)Kubh@)}(hSFor a complete listing of the indexed fields, please see the DataONE documentation.h]hSFor a complete listing of the indexed fields, please see the DataONE documentation.}(hjBhj@hhh'Nh)Nubah}(h]h]h]h]h ]uh&h?h'h(h)K hj/hhubh@)}(hLhttp://mule1.dataone.org/ArchitectureDocs-current/design/SearchMetadata.htmlh]h reference)}(hjPh]hLhttp://mule1.dataone.org/ArchitectureDocs-current/design/SearchMetadata.html}(hhhjTubah}(h]h]h]h]h ]refurijPuh&jRhjNubah}(h]h]h]h]h ]uh&h?h'h(h)K"hj/hhubh@)}(hIMetacat also reports on the currently-indexed fields, simply navigate to:h]hIMetacat also reports on the currently-indexed fields, simply navigate to:}(hjjhjhhhh'Nh)Nubah}(h]h]h]h]h ]uh&h?h'h(h)K$hj/hhubh@)}(hehttp://mule1.dataone.org/ArchitectureDocs-current/apis/MN_APIs.html#MNQuery.getQueryEngineDescriptionh]jS)}(hjxh]hehttp://mule1.dataone.org/ArchitectureDocs-current/apis/MN_APIs.html#MNQuery.getQueryEngineDescription}(hhhjzubah}(h]h]h]h]h ]refurijxuh&jRhjvubah}(h]h]h]h]h ]uh&h?h'h(h)K&hj/hhubh@)}(hwith "solr" as the engine.h]hwith “solr” as the engine.}(hjhjhhh'Nh)Nubah}(h]h]h]h]h ]uh&h?h'h(h)K(hj/hhubeh}(h]default-indexed-fieldsah]h]default indexed fieldsah]h ]uh&h*hh,hhh'h(h)Kubh+)}(hhh](h0)}(hIndex configurationh]hIndex configuration}(hjhjhhh'Nh)Nubah}(h]h]h]h]h ]uh&h/hjhhh'h(h)K+ubh@)}(hXKMetacat-index is deployed as a separate web application (metacat-index.war) and should be deployed as a sibling of the Metacat webapp (metacat.war). Deploying metacat-index.war is only required when SOLR support is desired (e.g., for MetacatUI) and can safely be omitted if it will not be utilized for any given Metacat deployment.h]hXKMetacat-index is deployed as a separate web application (metacat-index.war) and should be deployed as a sibling of the Metacat webapp (metacat.war). Deploying metacat-index.war is only required when SOLR support is desired (e.g., for MetacatUI) and can safely be omitted if it will not be utilized for any given Metacat deployment.}(hjhjhhh'Nh)Nubah}(h]h]h]h]h ]uh&h?h'h(h)K,hjhhubh@)}(hXDuring the initial installation/upgrade, an empty index will be initialized in the configured "solr-home" location. Metacat-index will index all the existing Metacat content when the webapp next initializes. Note: the configured solr-home directory should not exist before configuring Metacat with indexing for the first time, otherwise the blank index will not be created for metacat-index to utilize.h]hXDuring the initial installation/upgrade, an empty index will be initialized in the configured “solr-home” location. Metacat-index will index all the existing Metacat content when the webapp next initializes. Note: the configured solr-home directory should not exist before configuring Metacat with indexing for the first time, otherwise the blank index will not be created for metacat-index to utilize.}(hjhjhhh'Nh)Nubah}(h]h]h]h]h ]uh&h?h'h(h)K1hjhhubh@)}(hAdditional advanced configuration options are available in the metacat.properties file (shared between Metacat and Metacat-index).h]hAdditional advanced configuration options are available in the metacat.properties file (shared between Metacat and Metacat-index).}(hjhjhhh'Nh)Nubah}(h]h]h]h]h ]uh&h?h'h(h)K6hjhhubeh}(h]index-configurationah]h]index configurationah]h ]uh&h*hh,hhh'h(h)K+ubh+)}(hhh](h0)}(h+Adding additional document types and fieldsh]h+Adding additional document types and fields}(hjhjhhh'Nh)Nubah}(h]h]h]h]h ]uh&h/hjhhh'h(h)K:ubh@)}(hDTBD: Step-by-step guide for adding new documents and indexed fields.h]hDTBD: Step-by-step guide for adding new documents and indexed fields.}(hjhjhhh'Nh)Nubah}(h]h]h]h]h ]uh&h?h'h(h)K;hjhhubeh}(h]+adding-additional-document-types-and-fieldsah]h]+adding additional document types and fieldsah]h ]uh&h*hh,hhh'h(h)K:ubh+)}(hhh](h0)}(hQuerying the indexh]hQuerying the index}(hjhjhhh'Nh)Nubah}(h]h]h]h]h ]uh&h/hjhhh'h(h)K?ubh@)}(hThe SOLR index can be queried using standard SOLR syntax and return options. The DataONE query interface exposes the SOLR query engine.h]hThe SOLR index can be queried using standard SOLR syntax and return options. The DataONE query interface exposes the SOLR query engine.}(hj!hjhhh'Nh)Nubah}(h]h]h]h]h ]uh&h?h'h(h)K@hjhhubh@)}(hQhttp://mule1.dataone.org/ArchitectureDocs-current/apis/MN_APIs.html#MNQuery.queryh]jS)}(hj/h]hQhttp://mule1.dataone.org/ArchitectureDocs-current/apis/MN_APIs.html#MNQuery.query}(hhhj1ubah}(h]h]h]h]h ]refurij/uh&jRhj-ubah}(h]h]h]h]h ]uh&h?h'h(h)KChjhhubh@)}(hQPlease see the SOLR documentation for examples and exhaustive syntax information.h]hQPlease see the SOLR documentation for examples and exhaustive syntax information.}(hjGhjEhhh'Nh)Nubah}(h]h]h]h]h ]uh&h?h'h(h)KEhjhhubh@)}(hhttp://lucene.apache.org/solr/h]jS)}(hjUh]hhttp://lucene.apache.org/solr/}(hhhjWubah}(h]h]h]h]h ]refurijUuh&jRhjSubah}(h]h]h]h]h ]uh&h?h'h(h)KGhjhhubeh}(h]querying-the-indexah]h]querying the indexah]h ]uh&h*hh,hhh'h(h)K?ubh+)}(hhh](h0)}(hAccess Policy enforcementh]hAccess Policy enforcement}(hjxhjvhhh'Nh)Nubah}(h]h]h]h]h ]uh&h/hjshhh'h(h)KKubh@)}(hXAccess control is enforced by the index such that only records that are readable by the user performing the query are returned to the user. Any SOLR query submitted will be augmented with access control criteria corresponding to if and how the user is currently authenticated. Both certificate-based (DataONE API) and JSESSIONID-based (Metacat API) authentication are simultaneously supported.h]hXAccess control is enforced by the index such that only records that are readable by the user performing the query are returned to the user. Any SOLR query submitted will be augmented with access control criteria corresponding to if and how the user is currently authenticated. Both certificate-based (DataONE API) and JSESSIONID-based (Metacat API) authentication are simultaneously supported.}(hjhjhhh'Nh)Nubah}(h]h]h]h]h ]uh&h?h'h(h)KLhjshhubeh}(h]access-policy-enforcementah]h]access policy enforcementah]h ]uh&h*hh,hhh'h(h)KKubh+)}(hhh](h0)}(hRegenerating the indexh]hRegenerating the index}(hjhjhhh'Nh)Nubah}(h]h]h]h]h ]uh&h/hjhhh'h(h)KTubh@)}(hWhen the SOLR index has been drastically modified, a complete regeneration of the index may be necessary. In order to accomplish this:h]hWhen the SOLR index has been drastically modified, a complete regeneration of the index may be necessary. In order to accomplish this:}(hjhjhhh'Nh)Nubah}(h]h]h]h]h ]uh&h?h'h(h)KUhjhhubh)}(hhh](hc)}(h"Login as the Metacat administratorh]h@)}(hjh]h"Login as the Metacat administrator}(hjhjubah}(h]h]h]h]h ]uh&h?h'h(h)KXhjubah}(h]h]h]h]h ]uh&hbhjhhh'h(h)Nubhc)}(hBNavigate to: //metacat?action=reindexall h]h@)}(h?Navigate to: //metacat?action=reindexallh]h?Navigate to: //metacat?action=reindexall}(hjhjubah}(h]h]h]h]h ]uh&h?h'h(h)KYhjubah}(h]h]h]h]h ]uh&hbhjhhh'h(h)Nubeh}(h]h]h]h]h ]j"j#j$hj%j&uh&hhjhhh'h(h)KXubeh}(h]regenerating-the-indexah]h]regenerating the indexah]h ]uh&h*hh,hhh'h(h)KTubh+)}(hhh](h0)}(hClass design overviewh]hClass design overview}(hjhjhhh'Nh)Nubah}(h]h]h]h]h ]uh&h/hjhhh'h(h)K^ubhfigure)}(hhh](himage)}(hS.. figure:: images/indexing-class-diagram.png Figure 1. Class design overview. h]h}(h]h]h]h]h ]uri!images/indexing-class-diagram.png candidates}hjsuh&jhj h'h(h)Kbubhcaption)}(h Figure 1. Class design overview.h]h Figure 1. Class design overview.}(hj#hj!ubah}(h]h]h]h]h ]uh&jh'h(h)Kbhj ubeh}(h]id1ah]h]h]h ]uh&j h)Kbhjhhh'h(ubhcomment)}(hXu@startuml images/indexing-class-diagram.png package "Current cn-index-processor (library)" { interface IDocumentSubprocessor { + boolean canProcess(Document doc) + initExpression(XPath xpath) + Map processDocument(String identifier, Map docs, Document doc) } class AbstractDocumentSubprocessor { - List fields + setMatchDocument(String matchDocument) + setFieldList(List fieldList) } class ResourceMapSubprocessor { } class ScienceMetadataDocumentSubprocessor { } interface ISolrField { + initExpression(XPath xpathObject) + List getFields(Document doc, String identifier) } class SolrField { - String name - String xpath - boolean multivalue } class CommonRootSolrField { } class RootElement { } class LeafElement { } class FullTextSolrField { } class MergeSolrField { } class ResolveSolrField { } class SolrFieldResourceMap { } class SolrDoc { - List fieldList } class SolrElementField { - String name - String value } } IDocumentSubprocessor <|-- AbstractDocumentSubprocessor AbstractDocumentSubprocessor <|-- ResourceMapSubprocessor AbstractDocumentSubprocessor <|-- ScienceMetadataDocumentSubprocessor ISolrField <|-- SolrField SolrField <|-- CommonRootSolrField CommonRootSolrField o--"1" RootElement RootElement o--"*" LeafElement SolrField <|-- FullTextSolrField SolrField <|-- MergeSolrField SolrField <|-- ResolveSolrField SolrField <|-- SolrFieldResourceMap AbstractDocumentSubprocessor o--"*" ISolrField IDocumentSubprocessor --> SolrDoc SolrDoc o--"*" SolrElementField package "SOLR (library)" { abstract class SolrServer { + add(SolrInputDocument doc) + deleteByQuery(String id) + query(SolrQuery query) } class EmbeddedSolrServer { } class HttpSolrServer { } } SolrServer <|-- EmbeddedSolrServer SolrServer <|-- HttpSolrServer package "Metact-index (webapp)" { class ApplicationController { - List solrIndex + regenerateIndex() } class SolrIndex { - List subprocessors - SolrServer solrServer + insert(String pid, InputStream data) + update(String pid, InputStream data) + remove(String pid) } class SystemMetadataEventListener { - SolrIndex solrIndex + itemAdded(ItemEvent) + itemRemoved(ItemEvent) } } package "Metacat (webapp)" { class MetacatSolrIndex { - SolrServer solrServer + InputStream query(SolrQuery) } class HazelcastService { - IMap hzIndexQueue - IMap hzSystemMetadata - IMap hzObjectPath } } MetacatSolrIndex o--"1" SolrServer HazelcastService .. SystemMetadataEventListener ApplicationController o--"*" SolrIndex SolrIndex o--"1" SolrServer SolrIndex "1"--o SystemMetadataEventListener SolrIndex o--"*" IDocumentSubprocessor: Assembled using Spring bean configuration @endumlh]hXu@startuml images/indexing-class-diagram.png package "Current cn-index-processor (library)" { interface IDocumentSubprocessor { + boolean canProcess(Document doc) + initExpression(XPath xpath) + Map processDocument(String identifier, Map docs, Document doc) } class AbstractDocumentSubprocessor { - List fields + setMatchDocument(String matchDocument) + setFieldList(List fieldList) } class ResourceMapSubprocessor { } class ScienceMetadataDocumentSubprocessor { } interface ISolrField { + initExpression(XPath xpathObject) + List getFields(Document doc, String identifier) } class SolrField { - String name - String xpath - boolean multivalue } class CommonRootSolrField { } class RootElement { } class LeafElement { } class FullTextSolrField { } class MergeSolrField { } class ResolveSolrField { } class SolrFieldResourceMap { } class SolrDoc { - List fieldList } class SolrElementField { - String name - String value } } IDocumentSubprocessor <|-- AbstractDocumentSubprocessor AbstractDocumentSubprocessor <|-- ResourceMapSubprocessor AbstractDocumentSubprocessor <|-- ScienceMetadataDocumentSubprocessor ISolrField <|-- SolrField SolrField <|-- CommonRootSolrField CommonRootSolrField o--"1" RootElement RootElement o--"*" LeafElement SolrField <|-- FullTextSolrField SolrField <|-- MergeSolrField SolrField <|-- ResolveSolrField SolrField <|-- SolrFieldResourceMap AbstractDocumentSubprocessor o--"*" ISolrField IDocumentSubprocessor --> SolrDoc SolrDoc o--"*" SolrElementField package "SOLR (library)" { abstract class SolrServer { + add(SolrInputDocument doc) + deleteByQuery(String id) + query(SolrQuery query) } class EmbeddedSolrServer { } class HttpSolrServer { } } SolrServer <|-- EmbeddedSolrServer SolrServer <|-- HttpSolrServer package "Metact-index (webapp)" { class ApplicationController { - List solrIndex + regenerateIndex() } class SolrIndex { - List subprocessors - SolrServer solrServer + insert(String pid, InputStream data) + update(String pid, InputStream data) + remove(String pid) } class SystemMetadataEventListener { - SolrIndex solrIndex + itemAdded(ItemEvent) + itemRemoved(ItemEvent) } } package "Metacat (webapp)" { class MetacatSolrIndex { - SolrServer solrServer + InputStream query(SolrQuery) } class HazelcastService { - IMap hzIndexQueue - IMap hzSystemMetadata - IMap hzObjectPath } } MetacatSolrIndex o--"1" SolrServer HazelcastService .. SystemMetadataEventListener ApplicationController o--"*" SolrIndex SolrIndex o--"1" SolrServer SolrIndex "1"--o SystemMetadataEventListener SolrIndex o--"*" IDocumentSubprocessor: Assembled using Spring bean configuration @enduml}(hhhj8ubah}(h]h]h]h]h ]h$h%uh&j6hjhhh'h(h)Kubeh}(h]class-design-overviewah]h]class design overviewah]h ]uh&h*hh,hhh'h(h)K^ubeh}(h]metacat-indexingah]h]metacat indexingah]h ]uh&h*hhhhh'h(h)Kubeh}(h]h]h]h]h ]sourceh(uh&hcurrent_sourceN current_lineNsettingsdocutils.frontendValues)}(h/N generatorN datestampN source_linkN source_urlN toc_backlinksentryfootnote_backlinksK sectnum_xformKstrip_commentsNstrip_elements_with_classesN strip_classesN report_levelK halt_levelKexit_status_levelKdebugNwarning_streamN tracebackinput_encoding utf-8-siginput_encoding_error_handlerstrictoutput_encodingutf-8output_encoding_error_handlerjyerror_encodingUTF-8error_encoding_error_handlerbackslashreplace language_codeenrecord_dependenciesNconfigN id_prefixhauto_id_prefixid dump_settingsNdump_internalsNdump_transformsNdump_pseudo_xmlNexpose_internalsNstrict_visitorN_disable_configN_sourceh( _destinationN _config_files]pep_referencesN pep_base_url https://www.python.org/dev/peps/pep_file_url_templatepep-%04drfc_referencesN rfc_base_urlhttps://tools.ietf.org/html/ tab_widthKtrim_footnote_reference_spacefile_insertion_enabled raw_enabledKsyntax_highlightlong smart_quotessmartquotes_localesNcharacter_level_inline_markupdoctitle_xform docinfo_xformKsectsubtitle_xformembed_stylesheetcloak_email_addressesenvNgettext_compactubreporterNindirect_targets]substitution_defs}substitution_names}refnames}refids}nameids}(jSjPj,j)jjjjj jjpjmjjjjjKjHu nametypes}(jSNj,NjNjNj NjpNjNjNjKNuh}(jPh,j)hjj/jjjjjmjjjsjjjHjj1j u footnote_refs} citation_refs} autofootnotes]autofootnote_refs]symbol_footnotes]symbol_footnote_refs] footnotes] citations]autofootnote_startKsymbol_footnote_startKid_startKparse_messages]transform_messages] transformerN decorationNhhub.