Very Large Data Packages
========================
:Document Status:
======== ==================================================================
Status Comment
======== ==================================================================
DRAFT (rnahf) committed minor modifications shortly (1hr) after email
to developers@dataone.org
======== ==================================================================
.. contents::
Synopsis
~~~~~~~~~
While many data packages are of modest size (<100 objects), some large studies
generate upwards of 100,000 datasets that form a data package. These very large
data packages challenge performance limits in the DataONE data ingest cycle and
can present usability issues in user interfaces not prepared for them. Both memory
and processor time increase dramatically with increased number of data objects
and relationships expressed.
Potential submitters of packages containing large numbers of data objects must be
mindful that packages of such an large number of objects is likely to be unusable
for the majority of interested parties, and should consider consolidating and
compressing the individual objects into fewer objects to allow easier discovery/
inspection and download. This should be especially considered if the objects in
the package would not be usefully retrieved individually.
Creation of large resource maps is potentially the most time consuming activity,
depending on the tool used. Deserialization is comparatively quick, but the memory
requirements are high, depending on the type of model used during parsing. At the
stage of indexing, at issue is the time needed to process index record updates, as
well as the resulting number items in certain fields in the solr records. Last,
high-level client methods would like to safely be able to do whole-package downloads,
but need to be able to detect large data packages which could overwhelm their
ability to handle such as large package.
Below are discussions and test results of the known issues related to very large
resource maps, presented in order of when encountered in the object lifecycle.
Identified Issues
~~~~~~~~~~~~~~~~~
Resource map creation
---------------------
Use of the foresite library for building resource maps includes many checks to make
sure that the map validates. First the identifiers of the data and metadata are
added to a graph held in memory, then the graph is serialized to RDF/XML format.
For small packages the overhead for building the graph and performing consistency
checks is minimal, but both memory and time to build seem to scale geometrically
with the number of objects in the package.
Test results on different size resource maps are summarized below. In all cases
there is one metadata object that documents all of the objects.
============== ================ =========== ===========
# of objects time to build memory file size
============== ================ =========== ===========
10 7 K
33 20 K
100 2 seconds 45 MB 60 K
330 192 K
1000 6 seconds 20 MB 600 K
3300 24 seconds 23 MB 2 Mb
10000 4.5 minutes 30 MB 6 Mb
33000 66 minutes 142 MB 20 Mb
============== ================ =========== ===========
For creating very large resource maps, generation time using the java foresite
toolkit is an issue. Directly creating a serialized resource map is much faster.
For example, using an existing resource map as a template, and a short perl script,
a 100000 member resource map was created in approximately 10 seconds with the only
memory cost that of holding an identifier array in memory and any output buffering.
RDF Deserialization
---------------------
Deserialization happens both on the client side when downloading resource maps,
and on coordinating nodes, both when validating the resource map, and also when
indexing the relationships into the solr index. Performance metrics obtained
from JUnit tests monitored with Java Visual VM are summarized below. Fully
expressed resource maps were deserialized using both the default simple model,
and again using an OWL model loaded with the ORE schema to be able to do semantic
reasoning. The reasoning model adds an additional 268 triples from the ORE schema.
========== ======= ======== ======== ======= ======== ========
.. Default model Reasoning model
---------- --------------------------- ---------------------------
# objects triples time memory triples time memory
========== ======= ======== ======== ======= ======== ========
10 61 1 sec. 9 Mb 329 2 sec. 13 Mb
33 176 1 sec. 10 Mb 444 2 sec. 13 Mb
100 511 2 sec. 15 Mb 779 2 sec. 17 Mb
330 1661 2 sec. 20 Mb 1929 3 sec. 17 Mb
1000 5011 2 sec. 17 Mb 5279 3 sec. 24 Mb
3300 16511 3 sec. 20 Mb 16779 4 sec. 40 Mb
10000 50011 6 sec. 30 Mb 50279 8 sec. 90 Mb
33000 165011 7 sec. 51 Mb 165279 10 sec. 264 Mb
100000 500011 15 sec. 138 Mb 500279 26 sec. 792 Mb
========== ======= ======== ======== ======= ======== ========
The same information listed by model size shows that for small models, one can
see that memory requirements are not a simple function of number of triples, but
also a function of the model type. The reasoning model uses more memory per
triple than the simple model. Especially noticeable is that at very large sizes,
in terms of number of triples, the reasoning model uses significantly more memory.
======== ======= ======== ==========
triples time memory model type
======== ======= ======== ==========
61 1 sec. 9 Mb simple
176 1 sec. 10 Mb simple
329 2 sec. 13 Mb reasoning
444 2 sec. 13 Mb reasoning
511 2 sec. 15 Mb simple
779 2 sec. 17 Mb reasoning
1661 2 sec. 20 Mb simple
1929 3 sec. 17 Mb reasoning
5011 2 sec. 17 Mb simple
5279 3 sec. 24 Mb reasoning
16511 3 sec. 20 Mb simple
16779 4 sec. 40 Mb reasoning
50011 6 sec. 30 Mb simple
50279 8 sec. 90 Mb reasoning
165011 7 sec. 51 Mb simple
165279 10 sec. 264 Mb reasoning
500011 15 sec. 138 Mb simple
500279 26 sec. 792 Mb reasoning
======== ======= ======== ==========
The impact of this is that especially automated applications that deserialize RDF
files (such as the index processor) will need to be able to detect when they are
dealing with a resource map that could exceed available system resources.
It also seems wise, given that memory issues weigh larger than RDF file size,
to specify that resource maps with more than 50,000 triples need to fully express
relationships, instead of relying on reasoning models to infer semantically-defined
inverse relationships. This implies that if DataONE allows resource maps to sparsely
populate their relationships, that there also be tools to tell whether an RDF
is fully expressing relationships, or will be relying on semantic reasoning.
Indexing
---------
When resource maps are synchronized, the map is read and - once all of the package
members are indexed - the relationships in the map are added to the index records
of the data members. A 10000 member package will trigger the update of 10000
index records, adding the metadata object pid to the 'isDocumentedBy' field.
Additionally, both the 'contains' field in the resource map and the 'documents'
field in the metadata records will be updated with the pids of the 10000 members.
Such many-membered fields are difficult to impossible to display, and are time-
consuming to search when queried.
Indexing is by necessity a single-threaded process, one that can update on the
order of 100 records/minute. Therefore a package containing 100,000 members will
take about 1000 minutes, or about 17 hours. During this time, no other updates
will be processed.
Workarounds for this issue requires a redesign of the index processor so that the
large resource map does not delay other items in the indexing queue. Ultimately,
the solution would be to implement a different search engine for tracking package
relationships, and implementing another search endpoint using SPARQL
(http://en.wikipedia.org/wiki/SPARQL), and probably hiding the search query details
behind new DataONE API methods to spare the end user from having to learn another
query language to interact with DataONE.
Whole-Package Download
------------------------
The high-level DataPackage.download(packageID) method in d1_libclient implementations
by default downloads the entire collection of data package objects for local usage.
For these very-large data packages, the total package size is likely to be gigabytes
of information. In order to better support such convenience features, there needs
to be ways for determining the number of members of a package prior to download.
This would also help in situations where the number of package members is small,
but the individual data objects are large.
Mitigations
~~~~~~~~~~~~
It is useful for applications to know when a given data package is too large for
it to work with, or will require special handling. Ideally, this could be
determined before deserializing the xml, and even for some clients, prior to
download of the resource map itself.
Indexing performance is a function of member count, while deserialization performance
is a function of the number of triples. Download performance is a function of
total file size.
Determining Member Count
------------------------
For indexed resource maps, the easiest way to get the member count is with the
query::
cn/v1/query/solr/?q=resourceMap:{pid}&rows=0
For unindexed resource maps, the count of the number of occurences of the term
"ore:isAggregatedBy" in the RDF file will suffice.
Determining total package size for download
-------------------------------------------
To get the total size of the package, the following solr queries can be used::
# returns only sizes of package members
cn/v1/query/solr/?q=resourceMap:{pid}&fl=id,size
# returns sizes for package members and the resource map itself
cn/v1/query/solr/?q=resourceMap:{pid} OR id:{pid}&fl=id,size
from which the client could calculate the sum of the sizes returned.
To get the size of the resource map itself (useful for estimating memory requirements)::
# returns size of only the resource map
cn/v1/query/solr/?q=id:{pid}&fl=id,size
Determining Memory Requirements for deserialization
---------------------------------------------------
It is the number of triples and type of model used, moreso than the number of
package members, that best determines the graph model's memory requirement, and
so any additional triples expressed for each member would multiply the model size.
The use of ORE proxies, for example, or the inclusion of provenance information
are situations where this would be the case. DataONE *is* planning for the
inclusion of provenance statements in the resource maps, so users and developers
alike should take this into consideration.
The number of triples in an RDF/XML file can be determined either by parsing the
XML, or by estimating off the resource map byte count. By parsing the XML, one
would use an XML parser of choice to count all of the sub-elements of all of the
"rdf:Description" elements. In psuedo-code::
tripleCount = 0;
descriptionList = getRDFDescriptionElements();
foreach descriptionElement in descriptionList {
tripleCount += descriptionElement.getElementList().size;
}
To estimate from the file size, an upper limit of the number of triples can be deduced.
RDF/XML organizes triples as predicate-object sub-elements under an rdf:Description
element for each subject. If the ratio of subjects to triples is low, then the number
of bytes per triple is determined by the length of the predicate-object sub-element.
For a 30-character identifier, that sub-element is about 100 characters, and so::
upper limit on the number of triples = file size (bytes) / 100 bytes-per-triple
So for example, a 5Mb resource map has at most 50K triples, assuming an average
identifier size of 30 characters (URL encoded).
For a point of reference, a resource map for 1 metadata object documenting 1000
objects, expressing the 'ore:aggregates', 'ore:isAggregatedBy', 'cito:documents',
'cito:isDocumentedBy', and 'cito:identifier' predicates creates 5005 triples using
1003 subjects, and was tested to create 600K file. Applying the upper limit
approximation, (600K / 100 = 6K) gives 6000 triples, an over-estimate matching
the number of subjects.
Also note that long identifiers and identifiers predominated by non-ascii characters
that would be percent encoded in the file (3bytes per character) can lead to
an even higher upper limit than expected, and similarly, short identifiers in
the resource map could lead to a less robust upper limit.
Determining the memory requirement from the number of triples can be done either
by interpolating from the tables above, or by equation. Curve-fits of the
deserialization performance tests using polynomial equations gave the following::
simple model memory(Mb) ~ 2.6E-15 * triples^3 - 1.7E-09 * triples^2 + 0.00044 * triples + 12.7
(R2 = 0.99466)
reasoning model memory(Mb) ~ 1.25E-10 * triples^2 + 0.0015 * triples + 14.3
(R2 = 0.99997)
Note that the simple model required (rightly or wrongly) a third-order equation
to get a curve-fit with R2 > 0.9, whereas the reasoning model data could be highly
corelated with a binomial equation.
Expressed as a function of file size (bytes)::
simple model memory(Mb) ~ 2.6E-21 * size^3 - 1.7E-13 * size^2 + 4.4E-06 * size + 12.7
reasoning model memory(Mb) ~ 1.25E-14 * size^2 + 1.5E-05 * size + 14.3