GCP-CR Generation Challenge Programme Central Registry

Bioversity passport feature retrieval Moby workflow

This workflow of Moby web services allows clients to retrieve accession passport features from the databases managed by Bioversity International, namely the SINGER catalogue of CGIAR accessions, the EURISCO catalogue of European gene banks, a subset of the GRIN National Germplasm System and other eventual datasets, for a total of approximately 2 million accessions.

Although this service handle accession passport data, it can be applied to any other type of data, since the underlying principles of this workflow are general enough to be used as a template.

The workflow is constituted by four web services that will be called in sequence to retrieve accession passport features:

getAccessionFeatureOntologyTerms: This service can be used to retrieve the data dictionary, the result will be the list ofontology terms that the service understands.
countAccessionsByFeatures: This service receives a query, composed of conditions that use the terms provided by the first service and returns the count of the accession records that satisfy the query conditions.
getAccessionsByFeatures: This service receives the same input as the second service, and returns the list of accession identifiers that satisfy the query conditions.
getFeaturesByAccessionId: This last service receives the list of accession identifiers returned by the third service and returns the features of the accessions referred by the third service result.

These services are discoverable through IRRI's registry (http://cropwiki.irri.org/cgi-bin/MOBY-Central.pl) and you can test them using this interface.

getAccessionFeatureOntologyTerms

This service will return the data source dictionary, it will return the feature identifiers, their data type, the operators that can be used with them and the record count in the respective databases that hold those features.

Input

The input to this service is an empty Moby mobyData object.

Output

This output of service is a collection of GCP_SimpleIdentifierWithTypeOperatorAndCount Moby objects, each of which represent the information on a single feature within a single database.

.[@namespace]

The namespace refers to which database the feature belongs, registered namespaces currently are:

SINGER_Accession: The SINGER database (± 600,000 accessions).
EURISCO_Accession: The EURISCO database (± 1,100,000 accessions).
ALL_REASONABLE_Accession: All of the above, plus any other catalogue that is connected to the service. Currently there is a subset of the GRIN catalogue that contains ± 250,000 accessions and soon I will add the accessions referred to by the datasets in the Central Registry.

.[@id]

This element holds the feature identifier. These identifiers can be either an identifier of one of the Koios ontologies, or it can be an identifier of a private ontology, this value should be used whenever you want to refer to a feature.

These identifiers are unique, that is, you cannot have an identifier that refers to two different features or to a feature and an ontology term.

String[@articleName=”name”]/text()

This element holds the name or label of the feature.

String[@articleName=”operator”]/text()

This collection of elements holds the operators that can be used with the feature when creating query conditions, here are the valid codes:

EQ: Equals,
it is equivalent to A == B.
NE: Not equals,
it is equivalent to A != B.
AS: Like, it is equivalent to the SQL LIKE operator, it means case and accent insensitive compare. You can use the same wildcards as in SQL: % matches any number of characters, including zero characters; _ matches exactly one character.
It is equivalent to A LIKE B.
This operator can only be applied to string features.
NS: Not like, it is the negation of the previous operator,
it is equivalent to NOT( A LIKE B ).
This operator can only be applied to string features.
PX: Starts with,
it is equivalent to A LIKE ‘B%’.
This operator can only be applied to string features.
CX: Contains,
it is equivalent to A LIKE ‘%B%’.
This operator can only be applied to string features.
SX: Ends with,
it is equivalent to A LIKE ‘%B’.
This operator can only be applied to string features.
LT: Less than,
it is equivalent to A < B.
LE: Less than or equal,
it is equivalent to A <= B.
GT: Greater than,
it is equivalent to A > B.
GE: Greater than or equal,
it is equivalent to A >= B.
RG: Range, this operator matches values that fall within a range. The value must be expressed as a pair of sub-elements separated by a space character, such as in ‘120 300’ or ‘Anthony Michael’, the service will take care of selecting the lower and upper bound.
It is equivalent to the SQL BETWEEN A AND B.
NU: Empty or NULL, this operator can be used to match records that do not have the feature.
It is equivalent to the SQL A IS NULL.
NN: Not empty or NULL, this operator can be used to match records that do have the feature.
It is equivalent to the SQL A IS NOT NULL.
IN: Value matches at least one element of a list of values. This operator can be used by repeating the condition for each value in the list, or you can provide a list of values delimited by the first character of the string, such as ;10;20;30 in which the semicolon acts as the separator.
It is equivalent to the SQL X IN( 10, 20, 30 ).
NI: It is the negation of the previous operator.
It is equivalent to the SQL X NOT( IN( 10, 20, 30) ).
OT: Ontology term. In this case the dataType (see below) will be a term representing a leaf node of an ontology. Such terms must represent an element of an enumeration, as, for instance, Italy from a country ontology. The reason for this is to handle cases in which the codes referring to such elements can take different formats, as in the above example, where Italy is identified by any of these codes: 380, ITA and IT. If you receive such a feature, the last service will return the relevant identifier instead of the physical value stored in the database.

String[@articleName=”dataType”]/text()

This element represents the data type of the feature contents, the enumerated term referring to a single value that the feature holds, or the parent term of the feature. There are two basic types of data types:

Primitive type: A primitive data type defines the value contained in the feature, when providing a query condition to the other services, the condition value must be of one of these types
(note that the two trailing colons are part of the code):

string:: String or text data.
integer:: Integral numeric values.
float:: Floating point numbers.
date:: Dates, it is assumed that all dates are provided in this format: YYYY-MM-DD, in which YYYY represents the year, MM the month and DD the day.
Due to the nature of the data contained in the service’s data sources, dates will not be provided in this format, but in the string type (many dates lack the day and/or the month, so the standard format is not viable) following this format: YYYYMMDD, where MM and/or DD can be missing. This is not a problem since 19870101 will be smaller than 2000.
time:: Times should be provided in this format: HH:MM:SS.
dateTime:: This represents a time stamp and should be provided in this format: YYYY-MM-DD HH:MM:SS.
boolean:: A TRUE/FALSE flag, it should be provided as 1 (TRUE) or 0 (FALSE).
binary:: Binary data, it should be provided as a hexadecimal string such as 4D494C4B4F (MILKO).

Ontology term:In this case the data type contains an ontology term identifier. These terms should refer to an enumeration term, such as Italy in the examples above. Such data types will be repeated for each value that can be matched in the provided feature of the provided database. When providing a feature with this kind of data type, the last service will not return the physical value stored in the database, but it will return the term referring to that value, in this case the operator will always be OT. The feature that contains the controlled vocabulary will have as its data type the ontology term identifier of that controlled vocabulary.

String[@articleName=”count”]/text()

This element represents the number of records that the feature, or feature/data type combination has in the database indicated in the namespace element. This means that you may receive several Moby objects referring to the same feature in which only the namespace, data type and the count string values change. If a feature has no values in a specific database, the service will not return the Moby object describing the feature in that specific database.

Usage

The output of the service can be used to build web forms that allow querying the other services of this workflow, the returned elements follow these conventions:

The combination of feature identifier, name, operator and data type will be unique regardless of the namespace, this means that you can collect the data dictionary by selecting those elements.
If the feature has no values in any of the databases, the service will not return it. This means that the service does not return all the possible features, but only the existing features.
This also means that this service will return all the possible features: if you provide in the other services a feature identifier not returned by this service, the query will fail and the service will return a service exception listing the offending identifier.
You will not receive a feature that has both a primitive and term data type, this means that if you receive a primitive data type, the last service will always return a value of that type, if you receive a leaf term as the data type, the last service will return terms belonging to that ontology branch.
All features will be returned with their name or label, this means that you do not need to retrieve the name from the ontology; it also means that you may receive a feature belonging to a private ontology.
All ontology terms, those returned in the data type field, must belong to public ontology, since you will have to retrieve their name. This also means that if the database contains an extended set of enumerations, you will only be able to search for those belonging to the public ontology.
Martin, this is why I was mixing features and ontology terms…

In order to identify the type of feature, you have to look at the elements that compose the returned Moby objects, for this purpose we shall look at a series of real world examples.

Case A.

Suppose you receive the following Moby object:

<moby:GCP_SimpleIdentifierWithTypeOperatorAndCount
  moby:id="GCP_020:0000005"
  moby:namespace="SINGER_Accession">
  <moby:String moby:articleName="name">Genus</moby:String>
  <moby:String moby:articleName="operator">EQ</moby:String>
  <moby:String moby:articleName="operator">NE</moby:String>
  <moby:String moby:articleName="operator">PX</moby:String>
  <moby:String moby:articleName="operator">SX</moby:String>
  <moby:String moby:articleName="dataType">string</moby:String>
  <moby:String moby:articleName="count">1966945</moby:String>
</moby:GCP_SimpleIdentifierWithTypeOperatorAndCount>

The object provides the following information:

The namespace SINGER_Accession indicates that this feature can be found in the SINGER database.
The feature identifier is GCP_020:0000005, if this identifier can be found in the Koios ontologies, you can fetch additional information regarding it, if it doesn’t, this means that it belongs to a private ontology.
Or it means that someone changed the identifier in the source ontology and the change was not yet mirrored in the service data dictionary (like they did this week L).
The feature name is Genus, this name or label will be returned regardless if the feature is part of a public ontology or not.
The data type is string, this means that the feature holds text data. When providing query conditions to the other services, the condition value is expected to be a string and when the last service will return this feature values, these will be strings.
You can search for this feature by finding records in which the Genus is equal (EQ) to a string, is not equal to a string (NE), starts with a string (PX) or ends with a string (SX). The choice of these operators is in the hands of the data provider: you could search for genus containing a string, but in the majority of the cases you will look for genus that starts with a string and in some cases for those which end with a string.
The SINGER database has 1966945 records that feature a Genus.

This information would allow a web client to create an input with a popup menu featuring the operators and an input field that will hold the search value.

Case B.

Suppose you receive the following Moby objects:

<moby:GCP_SimpleIdentifierWithTypeOperatorAndCount
  moby:id="MyFeature"
  moby:namespace="SINGER_Accession">
  <moby:String moby:articleName="name">
    Biological status of accession code
  </moby:String>
  <moby:String moby:articleName="dataType">
   CO_020:0000075
  </moby:String>
  <moby:String moby:articleName="count">27444</moby:String>
</moby:GCP_SimpleIdentifierWithTypeOperatorAndCount>

<moby:GCP_SimpleIdentifierWithTypeOperatorAndCount
  moby:id="MyFeature"
  moby:namespace="SINGER_Accession">
  <moby:String moby:articleName="name">
    Advanced cultivar
  </moby:String>
  <moby:String moby:articleName="operator">OT</moby:String>
  <moby:String moby:articleName="dataType">
   CO_020:0000042
  </moby:String>
  <moby:String moby:articleName="count">12763</moby:String>
</moby:GCP_SimpleIdentifierWithTypeOperatorAndCount>

<moby:GCP_SimpleIdentifierWithTypeOperatorAndCount
  moby:id="MyFeature"
  moby:namespace="SINGER_Accession">
  <moby:String moby:articleName="name">
    Hybrid
  </moby:String>
  <moby:String moby:articleName="operator">OT</moby:String>
  <moby:String moby:articleName="dataType">
   CO_020:0000037
  </moby:String>
  <moby:String moby:articleName="count">7842</moby:String>
</moby:GCP_SimpleIdentifierWithTypeOperatorAndCount>

In this case the database is SINGER_Accession, the feature identifier is MyFeature which is a private ontology element of Biological status of accession type, it has 12763 advanced cultivar accessions and 7842 accession hybrids.
The first object defines the feature, in this case we provide the ontology term identifier of the controlled vocabulary in the data type (if the data source mirrors an existing ontology you could have the case in which the data type is the same as the fearure identifier: in my services this is the case.
The last two Moby objects provide the enumerations of the feature, along with their label, they have the feature identifier in their data type.
With this information you could build the following form input:

in which you would place the term identifiers received as data types as option codes and the names of the data types as the option labels.

countAccessionsByFeatures

This service will return the number of records that satisfy a collection of query conditions. The service receives a collection of Moby objects that constitute the query and will return an integer Moby object with the count of records matching the provided query.

Input

The input to this service is a collection of SimpleFeatureWithSearchOperator Moby objects that carry the following information:

.[@namespace]: The namespace refers to which database the provided condition should apply.
GCP_SimpleIdentifier[@id]: This represents the feature identifier of the subject of the condition.
String[@articleName=”operator”]: This represents the operator of the condition.
String[@articleName=”valueString”]: This represents the value of the condition.
String[@articleName=”name”]: This element is not used here.

Here are a couple of examples:

Case A.

<moby:GCP_SimpleFeatureWithSearchOperator moby:namespace="ALL_REASONABLE_Accession">
  <moby:String moby:articleName="operator">RG</moby:String>
  <moby:String moby:articleName="valueString">42 52</moby:String>
  <moby:GCP_SimpleIdentifier moby:id="CO_020:0000015" moby:articleName="type" />
</moby:GCP_SimpleFeatureWithSearchOperator>

This element will select:

in all databases (ALL_REASONABLE_Accession):

all latitudes (CO_020:0000015) of collecting site in the range (RG) of 42 to 52

Case B.

<moby:GCP_SimpleFeatureWithSearchOperator moby:namespace="EURISCO_Accession">
  <moby:String moby:articleName="operator">OT</moby:String>
  <moby:String moby:articleName="valueString"> CO_020:0000034</moby:String>
  <moby:GCP_SimpleIdentifier moby:id="MyFeature" moby:articleName="type" />
</moby:GCP_SimpleFeatureWithSearchOperator>

<moby:GCP_SimpleFeatureWithSearchOperator moby:namespace="EURISCO_Accession">
  <moby:String moby:articleName="operator">OT</moby:String>
  <moby:String moby:articleName="valueString"> CO_020:0000036</moby:String>
  <moby:GCP_SimpleIdentifier moby:id="MyFeature" moby:articleName="type" />
</moby:GCP_SimpleFeatureWithSearchOperator>

These elements will select:

in EURISCO (EURISCO_Accession):

all accessions whose biological status feature (MyFeature) is breeding material (CO_020:0000034) or synthetic population (CO_020:0000036)

Case C.

<moby:GCP_SimpleFeatureWithSearchOperator moby:namespace="EURISCO_Accession">
  <moby:String moby:articleName="operator">OT</moby:String>
  <moby:String moby:articleName="valueString"> CO_020:0000034</moby:String>
  <moby:GCP_SimpleIdentifier moby:id="MyFeature" moby:articleName="type" />
</moby:GCP_SimpleFeatureWithSearchOperator>

<moby:GCP_SimpleFeatureWithSearchOperator moby:namespace="EURISCO_Accession">
  <moby:String moby:articleName="operator">OT</moby:String>
  <moby:String moby:articleName="valueString"> CO_020:0000036</moby:String>
  <moby:GCP_SimpleIdentifier moby:id="MyFeature" moby:articleName="type" />
</moby:GCP_SimpleFeatureWithSearchOperator>

<moby:GCP_SimpleFeatureWithSearchOperator moby:namespace="EURISCO_Accession">
  <moby:String moby:articleName="operator">RG</moby:String>
  <moby:String moby:articleName="valueString">42 52</moby:String>
  <moby:GCP_SimpleIdentifier moby:id="CO_020:0000015" moby:articleName="type" />
</moby:GCP_SimpleFeatureWithSearchOperator>

These elements will select:

in EURISCO (EURISCO_Accession):

all accessions whose biological status (MyFeature) is breeding material (CO_020:0000034) or synthetic population (CO_020:0000036)
and all latitudes (CO_020:0000015) of collecting site in the range (RG) of 42 to 52

Case D.

<moby:GCP_SimpleFeatureWithSearchOperator moby:namespace="EURISCO_Accession">
  <moby:String moby:articleName="operator">OT</moby:String>
  <moby:String moby:articleName="valueString"> CO_020:0000034</moby:String>
  <moby:GCP_SimpleIdentifier moby:id="MyFeature" moby:articleName="type" />
</moby:GCP_SimpleFeatureWithSearchOperator>

<moby:GCP_SimpleFeatureWithSearchOperator moby:namespace="EURISCO_Accession">
  <moby:String moby:articleName="operator">OT</moby:String>
  <moby:String moby:articleName="valueString"> CO_020:0000036</moby:String>
  <moby:GCP_SimpleIdentifier moby:id="MyFeature" moby:articleName="type" />
</moby:GCP_SimpleFeatureWithSearchOperator>

<moby:GCP_SimpleFeatureWithSearchOperator moby:namespace="EURISCO_Accession">
  <moby:String moby:articleName="operator">RG</moby:String>
  <moby:String moby:articleName="valueString">42 52</moby:String>
  <moby:GCP_SimpleIdentifier moby:id="CO_020:0000015" moby:articleName="type" />
</moby:GCP_SimpleFeatureWithSearchOperator>

<moby:GCP_SimpleFeatureWithSearchOperator moby:namespace="SINGER_Accession">
  <moby:String moby:articleName="operator">RG</moby:String>
  <moby:String moby:articleName="valueString">900 1500</moby:String>
  <moby:GCP_SimpleIdentifier moby:id="CO_020:0000017" moby:articleName="type" />
</moby:GCP_SimpleFeatureWithSearchOperator>

<moby:GCP_SimpleFeatureWithSearchOperator moby:namespace="SINGER_Accession">
  <moby:String moby:articleName="operator">PX</moby:String>
  <moby:String moby:articleName="valueString">Tritic</moby:String>
  <moby:GCP_SimpleIdentifier moby:id="CO_020:0000005" moby:articleName="type" />
</moby:GCP_SimpleFeatureWithSearchOperator>

<moby:GCP_SimpleFeatureWithSearchOperator moby:namespace="SINGER_Accession">
<moby:String moby:articleName="operator">SX</moby:String>
  <moby:String moby:articleName="valueString">ale</moby:String>
  <moby:GCP_SimpleIdentifier moby:id="CO_020:0000005" moby:articleName="type" />
</moby:GCP_SimpleFeatureWithSearchOperator>

These elements will select:

in EURISCO (EURISCO_Accession):

all accessions whose biological status (MyFeature) is breeding material (CO_020:0000034) or synthetic population (CO_020:0000036)
and all latitudes (CO_020:0000015) of collecting site in the range (RG) of 42 to 52

or in SINGER (SINGER_Accession):

all records whose elevation (CO_020:0000016) ranges from 900 to 1500 meters
and all genus (CO_020:0000005) starting with Tritic or ending with ale

Condition joining.

Conditions are joined by the following rules:

Namespace: All conditions are first grouped by namespace, or database. These groups are joined by OR and the contents of these groups are parsed as follows:

Feature identifier: All feature identifiers within the same namespace are grouped together, these groups are joined by AND, the contents of these groups are parsed as follows:

Operator and value: All conditions are joined by OR.

So the above examples will resolve as follows:

Case A:
WHERE( (LATITUDE >= 42) AND (LATITUDE <= 52) )
Case B:
WHERE
(
(DATABASE = EURISCO) AND
(
(SAMPSTAT = 400) OR
(SAMPSTAT = 411)
)
)

Case C:
WHERE
(
(DATABASE = EURISCO) AND
(
    (SAMPSTAT = 400) OR
    (SAMPSTAT = 411)
) AND
(
    (LATITUDE >= 42) AND
    (LATITUDE <= 52)
)
)

Case D:
WHERE
(
(
    (DATABASE = EURISCO) AND
      (
      (SAMPSTAT = 400) OR
      (SAMPSTAT = 411)
      ) AND
      (
      (LATITUDE >= 42) AND
      (LATITUDE <= 52)
      )
) OR
(
      (DATABASE = SINGER) AND
      (
      (ELEVATION >= 900) AND
      (ELEVATION <= 1500)
      ) AND
      (
      (GENUS LIKE 'Tritic%') OR
      (GENUS LIKE '%ale')
      )
)
)

Output

This output of service is an Integer Moby object representing the global record count returned by the query. If you want separate counts for each database, you will have to provide one mobyData element per database and only use that database as namespace.

getAccessionsByFeatures

This service uses the same input as the previous service and, instead of returning the count of records that satisfy the query, it will return the collection of accession identifiers that match the query.

Input

The input to this service is the same as the input to countAccessionsByFeatures, so please refer to that service for an explanation.
Besides the mobyData collection, the service accepts the following parameters:

start: This parameter represents the record number from which the service should start returning records, it is equivalent to the SQL START operator.
limit: This parameter represents the number of records to be returned by the service, it is equivalent to the SQL LIMIT operator.

These parameters are essential because the service has a maximum amount of records it will return, by default this is 1024. This is necessary because this service can return almost 2,000,000 records, which is a lot of data to pack in a SOAP envelope and send through the Internet. You can use the previous service to retrieve the number of records, then use these parameters for paging through the results. If you provide a limit greater than the maximum allowed, the service will limit the output to its maximum and indicate this in a service exception.

Output

The output of this service is a collection of GCP_SimpleIdentifier, elements that hold the following information:

.[@namespace]: The namespace will hold the database namespace to which the accession belongs.
.[@id]: The id will hold the accession identifier in the database to which it belongs, this value will be meaningless for a human, but will have to be used by applications when requesting accession features in the last service. As a matter of fact, this value represents the primary key of the accession record, this value is immutable, provided that the elements comprising its LSID do not change.
String[@articleName=”name”]: This value will hold the LSID of the accession, due to the structure of this value, you can extract the following information.

Authority: The authority will be the FAO/WIEWS institute code of the gene bank holding the germplasm, you can download the database at http://apps3.fao.org/wiews/wiewspage.jsp?i_l=@@&show=DownloadinstEN.jsp, the value corresponds to the INSTCODE field. This database is updated daily, codes are nor deleted, nor reused, when institutions change name or affiliation, the old record is not deleted, but a new record is created and its INSTCODE is placed in the V_INSTCODE field of the old record.
Namespace: The namespace will be the collection code to which the accession belongs, for SINGER accessions it will be the germplasm collection, for EURISCO and the other accessions it will be their genus.
ObjectID: This value represents the Accession number, or catalogue number of the accession.
Version: This value, if present, will refer to the sample number of the accession. For the moment none of these databases divide accessions into samples, but as we will add Central Registry passport and when the SINGER data warehouse will be in place, it is possible that we also get this information.

Here is a snippet:

<moby:Simple>
  <moby:GCP_SimpleIdentifier
   moby:id="fb4a3f7476ca0b447bc8c6cb8930424d"
   moby:namespace="SINGER_Accession">
    <moby:String moby:articleName="name">
      urn:lsid:COL003:CIATBEAN:G20099
    </moby:String>
  </moby:GCP_SimpleIdentifier>
</moby:Simple>

<moby:Simple>
  <moby:GCP_SimpleIdentifier
   moby:id="9892847b5e8ca418e360db80d951636c"
   moby:namespace="EURISCO_Accession">
    <moby:String moby:articleName="name">
      urn:lsid:ESP004:Hordeum:NC017146
    </moby:String>
  </moby:GCP_SimpleIdentifier>
</moby:Simple>

The first accession is held in CIAT (COL003), belongs to the CIAT bean collection (CIATBEAN) and has an accession number of G20099.
The second accession is held in Instituto Nacional de Investigación y Tecnología Agraria y Alimentaria. Centro de Recursos Fitogenéticos in Madrid (ESP004), is a barley germplasm (Hordeum) and has an accession number of NC017146.
The first 4 elements cannot contain colons, the last one may.

getFeaturesByAccessionId

Provided the output of the previous service, this service will return the list of features relating to the accession identifiers provided as input.
This service represents the last step of the workflow, here is where you can get the passport information on accessions.

Input

The input to this service is the output of the previous service, without the parameters, please refer to that service for an explanation.
The input may have the following parameter: wantedFeatures, the value should be a comma delimited list of feature identifiers that you want to extract from the provided list of accessions; if you omit the parameter it is intended that you want all features.

Output

The output of the service will be a collection of GCP_SimpleFeature elements, each providing the following information:

.[@namespace]: The namespace refers to which database the provided accession belongs, the value refers to the registered namespace for that database.
.[@id]: The id refers to the accession identifier in the holding database, it will be its primary key, a persistent value.
String[@articleName=”name”]/text(): This element holds the same value returned in this field in the input message for that accession, in our case it will be the accession LSID.
String[@articleName=”valueString”]/text(): This element holds the value of the feature, it will either be a primitive type, if the first service defined that feature’s data type as a primitive type, or the ontology term corresponding to its value if in the first service the feature’s data type was defined as an ontology term.
If the returned value is a term, and that term does not belong to the Koios ontology, it means that it represents a term of a private ontology. This can occur if either the Koios ontology terms were modified, or if the database supports an extended set of enumerations.
GCP_SimpleIdentifier[@id]: This element holds the feature identifier.
GCP_SimpleIdentifier/String[@articleName=”name”]/text(): This element holds the feature name.

The order of the elements is not relevant.