ࡱ > g i f 7 ( bjbjUU &F 7| 7| b l Z Z Z Z \ N L ( t t t t < < < 9N ;N ;N ;N ;N ;N ;N $ O Q t _N < 9 . < < < _N @ Z Z t t E tN @ @ @ < Z ^ t t 9N @ < 9N @ @ H : eM , N M t @ `]94 = M
M L N 0 N M R @R ? * R M @ Z Z Z Z Building EcoGrid Query System
Version 1.0
Bing Zhu
San Diego Supercomputer Center
Jing Tao
National Center for Ecological Analysis and Synthesis
June 11, 2003
Table of Contents
EcoGrid Data Model Analysis
EcoGrid Query Tokenizer
Query Engines
1. EcoGrid Data Model Analysis
Within the current EcoGrid, two major data management systems have been used. They are Metacat [1] developed at National Center for Ecological Analysis and Synthesis and SRB [2] developed at San Diego Supercomputer Center.
SRB has been deployed to and has been used by scientific research projects such as Long Term Ecological Research project that uses SRB to store image files and user-defined metadata [3].
SRB stores data as files in a virtual hierarchical collections and sub-collections structure. Each file has its own system metadata such as owner, create date and file size. Thus the SRB system can be viewed as a virtual XML document that has collection(s) and file(s) as sub-nodes. The system metadata will serve as attributes for either collection or file. An extra attribute, object type, is needed SRB to indicate whether a node is a collection, a file, or a metadata. SRB also allow users to add their own metadata for each file. Each metadata will be a sub-node under a file or a collection.
Metacat is a network-enable metadata and data storage framework. It allows users store, query and retrieve XML documents with arbitrary schemas.
In Metacat, a XML file (Metadata) is treated as a document and it is stored in a relational database system. The entire documents can be thought as one collection of documents. When user searches data in Metacat, it will find all documents in the collection which match the search criteria and will return the document ids to the user.
Based on the above analysis of both SRB and Metacat, the EcoGrid data model is suitable for implementing either XML-based query suggested by Matt Jones, or XQuery, which is officially proposed and standardized by W3C community [4].
The Metacat already has an implementation of XML-based query for EML data models. The following is an example composed by Matt Jones.
Soils metadata query
%soil%
%soil%
%Jones%
%Vieglais%
Metacat has some JAVA code which can be revised to process the above MXL-Query document into a SQL query. The SQL query will be submitted to a RDBMS such as Oracle. Since SRBs MCAT is also running on top of a RDBMS, this implementation should be easily adopted by SRB.
This approach will provide a foundation for our implementation for our EcoGrid search system as we will describe it in details later.
Here we immediate face a question that what is the purpose of using XQuery.
One apparent reason is that it is not easy for user to construct their query into an XML document as described above. On the other hand, XQuery is a de-facto standard query language which is similar to SQL language and therefore will be very easy for software developers and any users with SQL knowledge to build their searches using XQuery.
As we are dealing with heterogeneous data sources in EcoGrid, we realized that there is a need for a common layer in our software stack for building a query system within EcoGrid. This layer should be designed and implemented in a way that can be easily integrated with either one of the data management systems. And using XQuery as a common search language in SEEK EcoGrid will serve this purpose.
The XQuery acts as a language-neutral implementation layer for underneath heterogeneous systems such as Metacat and SRB. The EcoGrid Tokenizer is designed to translate the XQuery input into XM-Query document which can be consumed by both Metacat and SRB search engine.
The following diagram illustrates the overall search flow within the current EcoGrid.
2. EcoGrid Query Tokennizer
The EcoGrdi Query Tokenizer is one of the two key components in our overall implementation. (Another one is the query engine.)
Our tokenizer is designed in our search system to translate XQueries into XML query documents as we shown in the example in EcoGrid Data Model Analysis.
When our Tokenizer receives an XQuery from a user or an application, it will decompose the XQuery input and create object(s) each of which has a list of combination of information such as action mode, value, and parameters.
The following is an example of XQuery search on EcoGrid for Metacat.
declare namespace eml=eml://ecoinformatics.org/eml-2.0.0
for $e in input()
where $e//dataset/creator/surName=Johnson and
$e//dataset/pubDate=09-23-2002
return $e
When the EcoGrid Query Toeknizer receives the above query, it decomposes the query into a top query object which has child objects of Term and an attribute, Relation, to indicate the database operation. In the following diagram, Term1 and Term2 have Intersect (AND) relationship (Union will be OR relationship).
For more complicated case, an object can be embedded into another object. So the query data structure has the ability to handle subquery.
SHAPE \* MERGEFORMAT
Figure 3 Objects Embedded in Another Object
After all objects are constructed, they will be serialized into a XML Query document by Toeknizer. The XML Query document will then be sent to both Metacat and SRB search engines.
In building object(s) for underneath systems, the Tokennizer needs to query the EcoGrid registry to dynamically find the search nodes from the XML-Query document or an XQuery statement. One example is that it has to find the search nodes in the following XQuery example.
declare namespace xlink = "http://www.w3.org/1999/xlink"
{
for $hr in input()//@xlink:href
return { $hr }
}
Once the object(s) is/are constructed, it/they will be serialized into an XML-Query document which then will be submitted to both Metacat and SRB search engines.
3. Query Engines
Both Metacat team and SRB team are responsible for building Query engines to translate XML-Query document input from Tokenizer into native queries for either system. The search results will be simply a list of document names from metacat and a list of file names with collection paths in SRB.
The search in SRB is mainly performed on system metadata and user-defined metadata. Since SRB allows users to add any metadata with format of attribute-value-[unit] for a file, the SRB Query engine will translate the SQL statements into a local search format for SRB APIs (similar to Smeta) which in turn will be translated into SQL statement by a SRB server and will be submitted to the RDBMS in which SRB MCAT tables are installed.
Currently Metacat has an existing implementation for similar XML-Query document. The implementation only need be revised slightly.
The following is an example of the input and output of Metacat Query Engine in processing an XQuery statement.
Reference
[1] Metacat: the flexible XML database,
http://knb.ecoinformatics.org/software/metacat
[2] Storage Resource Broker, http://www.sdsc.edu/DICE/SRB.
[3] San Diego Supercomputer Center Releases New Version of Widely Used Data.
Management Software , http://www.sdsc.edu/Press/03/031003_srb3.html.
[4] XML Query, http://www.w3.org/XML/Query.
SHAPE \* MERGEFORMAT
Tokenizer
Metacat Query engine
SRB Query engine
Query Front-end
Query for Metacat
Query for SRB
XQuery
Figure 2 Construct an Object in Tokenizer
Object3
Relation: Union
EcoGrid Registry
Figure 1. Query procedure in EcoGrid
Metacat data source
SRB Data source
Object1
SQL
Result Set
declare namespace eml=eml://ecoinformatics.org/eml-2.0.0
for $e in input()
where $e//dataset/creator/surName=Johnson and
$e//dataset/pubDate=09-23-2002
return $e
Metacat Native Query Engine
(Parsing XML and Constructing SQL)
XML-Query Document
Client
Metacat
Data Source
XML-Query Document
Tokenizer
Object1
Relation: Intersect
Term1
Param: dataset/creator/surname
Value: Johnson
Mode: equals
Term2
Param: dataset/pubDate
Value: 09-23-2002
Mode: equals
Object2
Figure 5 Tokenizer and Metacat Search Engine
Result Set
p + , - . 1 2 3 4 6 7 ? P U V M N e f g i " " $ $ $ $ $ $ $ $ $ $ $ $ $ % % % % % % ^% o% % % % % % & ' Ƶ CJ aJ (CJ OJ PJ QJ ^J aJ mH%nH sH%tH CJ !j B*UmH nH ph tH u j U B*ph * B*ph %j B*CJ UmH nH ph tH u *
j CJ UmH nH tH u mH%sH%< * + 4 S T ]
& F h^h $a$ $a$ $
(
e f E
C = n 1 2 ' ( ) * + - / 1 3 5 6 8 9 : ; < =
` = > ? Q R S T U W X Y u v q 4 U t u M j k M N ] ^ 0 A V W $a$ W 2 3 ! ! o" p" " " " " " " " " " " " " " " " " " " $a$ " " " " " " " " " " " " " " " # # # # # # # # # #
# # #
# # $a$ # # # # # # # # # # # # # # %# &# Q# # # $ a$ $ $ $ $ $ $ $ $ $a$ $ $ $ $ % % % % % % % D% E% M% ]% ^% o% p% % % % % % % % % % % $7$ 8$ H$ a$ $a$ % % % % & %& U& v& & & & & & & & & & & & & ' ' ' ' '' (' 0' D' E' $a$ ' ' ( CJ E' K' j' y' ' ' ' ' ' ' ' ' ' ' ' ' ( ( (
( ( $a$ $ &P 1h / =!"#$% D d +q D
3 @@ " ?
i < @ < N o r m a l CJ _HaJ mH nHsH tH0 @ 0 H e a d i n g 1 $@& >* < A@ < D e f a u l t P a r a g r a p h F o n t . >@ . T i t l e $a$
5CJ \ L@ D a t e . U@ . H y p e r l i n k >*B*ph ! 3 D W f o ! - - A I ^ r } # $ T U a $
t p o z D F f g h ! 3 D W f o ! - - A I ^ r } # $ T U a d
$ F * + 4 S T ] e f E C
= n
1 2 ' ( ) * + - / 1 3 5 6 8 9 : ; < = > ? Q R S T U W X Y u v q 4 U t u M j k M N ] ^ 0 A V W 2 3 o p
% & Q a ! ! ! ! ! ! ! D! E! M! ]! ^! o! p! ! ! ! ! ! ! ! ! ! ! ! ! ! " %" U" v" " " " " " " " " " " " " # # # # '# (# 0# D# E# K# j# y# # # # # # # # # # # # # $ $ $ $ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Y 0 Y 0 Y 0 Y 0 Y 0 Y 0 Y 0 Y 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ' ( ! = W " # $ % E' ( "
( M e h $ 8 @ < 0 (
F " !
# s "* ? `
c $ X 9 9 ? " ! h
3 " ` \
c H! 9!
h
3 " ` P X
h
3 " ` L T
\ p %
s # " h
f
3 " ` p %
n
g
C g " ` 5 3
h
h
3 " ` 3 c$
`
o
c $
r
6 .
Z
S ̙ (
Z
S $
T B
@
c $ D # r
6
T B
c $ D B b
3 B b
3 r
6
/
\ ' + o
x # " ~
6 ' + '
` B
c $ D Q Q ` B
c $ D = = o \ " _(
w # " ~
6 " m _( q 6
` B
c $ D q% q% ' " ` B
c $ D % % l
0
l
0
T B
c $ D T B
l
c $ D
T B
m
c $ D f
p
s *
p
Z B
q
s * D f
t
s * t
N B
u
S D N B
v
S D N
3
N B
S D
B @ p ( *-
~ L p ( p)
p ( p) t
D
S D " ` P
X 2
\ b
E
3 " ` ! @ p) 0 t
F
S F " ` 8 $ h( 7
T
z
# z | `
T B
{
C D } Z B
| B
S D K! T B
B
C D % % Z B
S D % n
C " ` ( J
Z B
S D c o
0 " ` %
n
C " ` p
n
C " ` 9
h
3 " ` @ h 9
Z B
S D
$ Z B
S D
h
3 " ` |
!
l
c $ z
+ v *-
B
S ? + - 1 3 6 ? @ A B C D E F G H I J K L M N O U f $ p ~
^ t q
Z Z t }
D t X y t p \ t $ $ t
t v { N{ dt u 0 t x t 8 8 t l t $ =t l p p t x v 5t w g_ t m ] ] t
M t t m q t h v D t 3 t o H t s Y i
t t ~ # } t
t ! b% t * q E T
C
[
= T n ! ! $ p D S
B
Z
< S m 5 6 U [ n u ! ! ! ! " " %" *" V" W" v" |" # # $ 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 O ! ! ! ! ! E! \! ^! n! ! ! ! ! ! ! ! ! ! ! " " " " " # # # # &# (# C# # # # # $
$ $ B i n g Z h u W C : \ R & D \ S E E K \ s e e k _ c v s \ s e e k \ p r o j e c t s \ e c o g r i d \ d o c s \ Q u e r y I n t e r f a c e \ q u e r y T o k e n i z e r E n g i n e . d o c B i n g Z h u W C : \ R & D \ S E E K \ s e e k _ c v s \ s e e k \ p r o j e c t s \ e c o g r i d \ d o c s \ Q u e r y I n t e r f a c e \ q u e r y T o k e n i z e r E n g i n e . d o c B i n g Z h u W C : \ R & D \ S E E K \ s e e k _ c v s \ s e e k \ p r o j e c t s \ e c o g r i d \ d o c s \ Q u e r y I n t e r f a c e \ q u e r y T o k e n i z e r E n g i n e . d o c B i n g Z h u W C : \ R & D \ S E E K \ s e e k _ c v s \ s e e k \ p r o j e c t s \ e c o g r i d \ d o c s \ Q u e r y I n t e r f a c e \ q u e r y T o k e n i z e r E n g i n e . d o c B i n g Z h u W C : \ R & D \ S E E K \ s e e k _ c v s \ s e e k \ p r o j e c t s \ e c o g r i d \ d o c s \ Q u e r y I n t e r f a c e \ q u e r y T o k e n i z e r E n g i n e . d o c B i n g Z h u W C : \ R & D \ S E E K \ s e e k _ c v s \ s e e k \ p r o j e c t s \ e c o g r i d \ d o c s \ Q u e r y I n t e r f a c e \ q u e r y T o k e n i z e r E n g i n e . d o c B i n g Z h u W C : \ R & D \ S E E K \ s e e k _ c v s \ s e e k \ p r o j e c t s \ e c o g r i d \ d o c s \ Q u e r y I n t e r f a c e \ q u e r y T o k e n i z e r E n g i n e . d o c B i n g Z h u W C : \ R & D \ S E E K \ s e e k _ c v s \ s e e k \ p r o j e c t s \ e c o g r i d \ d o c s \ Q u e r y I n t e r f a c e \ q u e r y T o k e n i z e r E n g i n e . d o c B i n g Z h u W C : \ R & D \ S E E K \ s e e k _ c v s \ s e e k \ p r o j e c t s \ e c o g r i d \ d o c s \ Q u e r y I n t e r f a c e \ q u e r y T o k e n i z e r E n g i n e . d o c B i n g Z h u l C : \ D o c u m e n t s a n d S e t t i n g s \ b i n g \ A p p l i c a t i o n D a t a \ M i c r o s o f t \ W o r d \ A u t o R e c o v e r y s a v e o f q u e r y T o k e n i z e r E n g i n e . a s d Eb z4)
^`o( h H [ ]
^`h H .
pL p^p`Lh H .
@ @ ^@ `h H .
^`h H .
L ^`Lh H .
^`h H .
^`h H .
PL P^P`Lh H . ^`o( . ^` . pL p^p`L . @ @ ^@ ` . ^` . L ^`L . ^` . ^` . PL P^P`L . z Eb 4 D# # # $ $ $ a0 a0a0a0a0a0@ * * G * * $ @ @ U n k n o w n G z T i m e s N e w R o m a n 5 S y m b o l 3& z A r i a l ; B a t a n g " h {vv&
9 ; ! x 20 d !
3Q X D a t e : J u n e 1 0 , 2 0 0 3 B i n g Z h u B i n g Z h u F Z Z ZZ pm06_10_2003 withJing_revise.doc # # Oh +'0
< H
T ` h p x Date: June 10, 2003 ate
Bing Zhu 1 Bin Bin Normal.dot 1
Bing Zhu 1 27 n Microsoft Word 9.0 @ _ @ L3@ 2X94 ՜. +,0 h p |
9
!
Date: June 10, 2003 Title
! " # % & ' ( ) * + - . / 0 1 2 3 4 5 6 7 8 9 : ; < = > ? @ A B C D E F G H I J K L M N O P Q R S T U W X Y Z [ \ ] _ ` a b c d e h R o o t E n t r y F 0/]94j D a t a
$ 1 T a b l e , R W o r d D o c u m e n t &F S u m m a r y I n f o r m a t i o n ( V D o c u m e n t S u m m a r y I n f o r m a t i o n 8 ^ C o m p O b j j O b j e c t P o o l 0/]940/]94
F Microsoft Word Document
MSWordDoc Word.Document.8 9q