In the Virtual Observatory (VO), registries provide a means for
discovering useful resources, i.e., data and services. Individual
publishers offer the descriptions for their resources ("resource
records") in publishing registries. At the time of writing, there are
roughly 14000 such resource records active within the VO, originating
from about 20 publishing registries.
The protocol spoken by these
publishing registries, OAI-PMH only allows restricting queries by
modification date and identifier and is hence not suitable for data discovery.
Even if it were, data discovery would at least be fairly time consuming if
each client had to query dozens or, potentially, hundreds of
publishing registries.
To enable efficient data discovery nevertheless, there are services harvesting the
resource records from the publishing registries and offering rich query
languages.
The IVOA Registry
Interfaces specification std:RI1 defined, among other aspects of
the VO registry system, such an interface
using SOAP and an early draft of an XML-based query language.
This document provides an update to the query ("searchable registry") part
of that specifiation (chapter 2), aimed towards
usage with current VO standards, in particular TAP std:TAP
and ADQL std:ADQL. It follows the model of ObsCore
std:OBSCORE of defining a representation of a data model
within a relational database. In this case, the data model is a
simplification of the VO's resource metadata interchange representation,
the VOResource XML format std:VOR. The simplification
yields 13 tables. For each table, TAP_SCHEMA metadata is given together
with rules for how to fill these tables from VOResource-serialized
metadata records as well as conditions on foreign keys and
recommendations on indices.
The resulting set of tables has a modest size by today's standards,
but is still non-trivial. The largest table, table_column,
has about half a million rows at the time of writing.
The architecture laid out here allows client applications to perform "canned"
queries on behalf of their users as well as complex queries formulated
directly by advanced users, using the same TAP clients they employ to
query astronomical data servers.
The combination of the table set discussed here and a TAP server is
referred to in this document as "RegTAP".
This specification directly relates to other VO standards in the
following ways:
- VOResource, v1.03 std:VOR
- VOResource sets the foundation for a formal definition of the data
model for resource records via its schema definition. This document
refers to concepts laid down there via XPaths std:XPATH.
- VODataService, v1.1 std:VODS11
- VODataService extends the VOResource data model by important
concepts and resource types (tablesets, data services, data
collections). These concepts and types are reflected in the database
schema. Again xpaths link this specification and VODataService.
- Other Registry Extensions
- Registry extensions are VO standards
defining how particular resources (e.g., Standards) or capabilities
(e.g., IVOA defined interfaces) are described. Most aspects
introduced by them are reflected in the
res_detail table using
xpaths into the registry documents.
This document should not in general need updates
for registry extension updates. For completeness, we note the
versions current as of this specification: SimpleDALRegExt 1.0
std:DALREGEXT,
StandardsRegExt 1.0 std:STDREGEXT, TAPRegExt 1.0
std:TAPREGEXT, Registry Interfaces 1.0 std:RI1.
- TAP, v1.0 std:TAP
- The queries against the schema defined in the present document, and the results of
these queries, will usually be transported using the Table Access
Protocol TAP. It also allows discovering
local additions to the registry relations via TAP's metadata publishing
mechanisms.
- IVOA Identifiers, v1.12 std:VOID
- IVOA identifiers are something like the primary keys within the VO
registry; as such, they are actual primary keys of the central table of
the relational registry. Also, the notion of an authority as laid down
in IVOA Identifiers plays an important role as publishing registries can
be viewed as a realization of a set of authorities.
This standard also relates to other IVOA standards:
- ADQL, v2.0 std:ADQL
- The rules for ingestion are designed to allow
easy queries given the constraints of ADQL 2.0. Also,
we give three functions that extend ADQL using the
language's built-in facility for user-defined functions.
- RegTAP-STC
- This specification will be complemented by a schema of four tables, also
under the
rr schema making up the relational registry, that gives coverages of resources on the spatial,
temporal, spectral and redshift axes. These will be defined in a
separate document.
In the design of the tables, the goal has been to preserve as much of
VOResource and its extensions, including the element names, as
possible.
An overriding consideration has been, however, to make natural joins
between the tables behave usefully, i.e., to actually combine rows
relevant to the same entity (resource, table, capability, etc.).
To disambiguate column names that name the same concept on different
entities (name, description, etc.) and would therefore interfere with
the natural join, a shortened tag for the source object
is prepended to the name. Thus, a description element within
a resource ends up in a column named
res_description, whereas the same element from a
capability becomes cap_description.
We further renamed some columns with respect to their VOResource
counterparts to avoid clashes with reserved words in popular database
management systems. The alternatives would have been to either recommend
quoting them or burden ADQL translation layers with the task of
automatically converting them to delimited identifiers. Both
alternatives seemed either confusing or impacting robustness.
Furthermore, camel-case identifiers have been converted to
underscore-separated ones (thus, standardId becomes
standard_id) to have all-lowercase column names; this saves
potential headache if users choose to reference the columns using SQL
delimited identifiers. Dashes in VOResource attribute names are
converted to underscores, too, with the exception of
ivo-id, which is just rendered ivoid.
Another design goal of this specification has been that different registries
operating on the same set of registry records will return identical responses
for most queries; hence, we try to avoid relying on features left not
defined by ADQL (e.g., the case sensitivity of string matches). However,
with a view to non-uniform support for information retrieval-type
queries in database systems, the ivo_hasword user defined
function is not fully specified here; queries employing it may yield
different results on different implementations, even if they operate on
the same set of resource records.
This specification piggybacks on top of the well-established
VOResource standard. This means that it does not define a full data model,
but rather something like a reasonably query-friendly view of a partial
representation of one. The link between the actual data model, i.e.,
VOResource and its extensions as defined by the XML Schema documents, and
the fields within this database schema, is provided by
xpaths, which are here slightly abbreviated for both brevity and
generality.
All xpaths given in this specifiation are assumed to be relative to
the enclosing vr:Resource element; these are called
"resource xpaths" in the following. If resource xpaths are to be
applied to an OAI-PMH response, the xpath expression
*/*/*/oai:metadata/ri:Resource must be prepended to it,
with the canonical prefixes from section qnameatts implied. The resource xpaths themselves
largely do not need explicit namespaces since VOResource elements are by
default unqualified. Elements and attributes from non-VOResource
schemata have the canonical namespace prefixes, which in this
specification only applies to several xsi:type attribute
names.
Some tables draw data from several different VOResource elements.
For those, we have introduced an extended syntax with additional
metacharacters (, ), and |, where the vertical bar denotes an
alternative and the parentheses grouping. For instance, our notation
/(tableset/schema/|)table/ corresponds to the two xpaths
/table and /tableset/schema/table.
Within the Virtual Observatory, the link between data models and
concrete data representations is usually made using utypes.
Since VOResource is directly modelled
in XML Schema, the choice of XPath as the bridging formalism is
compelling, though, and utypes themselves are not necessary for the
operation of a TAP service containing the relational registry.
TAP, however, offers fields for utypes in its TAP_SCHEMA. Since they
are not otherwise required, this specification takes the liberty of
using them to denote the xpaths.
In the metadata for tables and columns below, the utypes given are
obtained from the xpaths by simply prepending them with
xpath:. To avoid repetition, we allow relative xpaths:
when the xpath in a column utype does not start with a slash, it is
understood that is must be concatenated with the table utype to obtain
the full xpath.
For illustration, if a table has a utype of
xpath:/capability/interface/ and a column within this table
has a utype of xpath:accessURL/@use, the resulting resource
xpath would come out to be
/capability/interface/accessURL/@use; to match this in an
OAI-PMH response, the xpath would be
*/*/*/oai:metadata/ri:Resource/capability/interface/accessURL/@use.
While clients MUST NOT rely on these utypes in either TAP_SCHEMA or the
metadata delievered with TAP replies, service operators SHOULD provide them, in
particular when there are local extensions to the relational registry in their
services. Giving xpaths for extra columns and tables helps human
interpretation of them at least when the defining schema files are
available.
Resource xpaths are also used in the res_details table (section table_res_detail). These are normal xpaths
(although again understood relative to the enclosing Resource element),
which, in particular, means that they are case sensitive. On the other
hand, to clients they are simply opaque strings, i.e., clients cannot
just search for any xpaths into VOResource within res_details.
In the following table descriptions, the names of tables and columns
are normative and MUST be used as given, and all-lowercase. On the
values in the utype columns within TAP_SCHEMA,
see section vorutypes. All columns defined in
this document MUST have a 1 in the std column of the
TAP_SCHEMA.table_columns table. Unless otherwise
specified, all values of ucd and unit in
TAP_SCHEMA.table_columns are NULL for columns defined here.
Descriptions are not normative (as given, they usually are taken from
the schema files of VOResource and its extensions with slight
redaction). Registry operators MAY provide additional columns in their
tables, but they MUST provide all columns given in this
specification.
All table descriptions start out with brief remarks on the
relationship of the table to the VOResource XML data model. Then, the
columns are described in a selection of TAP_SCHEMA metadata. For each
table, recommendations on explicit primary and foreign keys as well as
indexed columns are given, where it is understood that primary and
foreign keys are already indexed in order to allow efficient joins;
these parts are not normative, but operators should ensure decent
performance for queries assuming the presence of the given indices and
relationships. Finally, lowercasing requirements
(normative) are given.
The following tables make up the data model
ivo://ivoa.net/std/RegTAP/vor:
Table
Utype | Description |
|---|
rr.capability
xpath:/capability/ | Pieces of behaviour of a resource. |
rr.interface
xpath:/capability/interface/ | Information on access modes of a capability. |
rr.intf_param
xpath:/capability/interface/param/ | Input parameters for services. |
rr.relationship
xpath:/content/relationship/ | Relationships between resources, e.g., mirroring, derivation, but
also providing access to data within a resource. |
rr.res_date
xpath:/curation/ | A date associated with an event in the life cycle of the resource.
This could be creation or update. The role column can be used to
clarify. |
rr.res_detail
N/A | XPath-value pairs for members of resource or capability and their
derivations that are less used and/or from VOResource extensions. The
pairs refer to a resource if cap_index is NULL, to the referenced
capability otherwise. |
rr.res_role
N/A | Entities, i.e., persons or organizations, operating on resources:
creators, contacts, publishers, contributors. |
rr.res_schema
xpath:/tableset/schema/ | Sets of tables related to resources. |
rr.res_subject
xpath:/content/ | Topics, object types, or other descriptive keywords about the
resource. |
rr.res_table
xpath:/(tableset/schema/|)table/ | (Relational) tables that are part of schemata or resources. |
rr.resource
xpath:/ | The resources, i.e., services, data collections, organizations, etc.,
present in this registry. |
rr.table_column
xpath:/(tableset/schema|)/table/column/ | Metadata on columns of tables pertaining to resources. |
rr.validation
xpath:/(capability|) | Validation levels for resources and capabilities. |
The rr.resource table contains most atomic members of
vr:Resource that have a 1:1 relationship to the resource
itself. Members of derived types are, in general, handled through
the res_detail
table even if 1:1 (see table_res_detail). The
content_level, content_type, waveband, and rights members are 1:n but still appear
here. If there are multiple values, they are concatenated with hash
characters (#). Use the ivo_hashlist_has ADQL extension
function to check for the presence of a single value. This convention
saves on tables while not complicating common queries significantly.
A local addition is the creator_seq column. It contains
all content of the name elements below a resource element
curation child's creator children,
concatenated with a sequence of semicolon and blank characters (";
"). The individual parts must be concatenated preserving the
sequence of the XML elements. The resulting string is primarily intended
for display purposes ("author list") and is hence not case-normalized.
It was added since the equivalent of an author list is expected to be
a metadatum that is displayed fairly frequently, but also since the
sequence of author names is generally considered significant. The
res_role table, on the other hand, does not allow
recovering the input sequence of the rows belonging to one resource.
The res_type column reflects the lower-cased value of
the ri:Resource element's xsi:type attribute,
where the canonical prefixes are used. While custom or experimental
VOResource extensions may lead to more or less arbitrary strings in that
column, VOResource and its IVOA-recommended extensions at the time of
writing define the following values for res_type:
- vg:authority
- A naming authority (these records allow resolving who is responsible
for IVORNs with a certain authority; cf. std:VOID)
- vg:registry
- A registry is considered a publishing registry if it contains a
capability element with
xsi:type="vg:Harvest". Old,
RegistryInterface 1-compliant registries also use this type with a
capability of type vg:Search. The relational registry as
specified here, while superceding these old vg:Search
capabilities, does not use this type any more. See section
registration on how to locate services
supporting it.
- vr:organisation
- The main purpose of an organisation as a registered resource is to
serve as a publisher of other resources.
- vr:resource
- Any entity or component of a VO application that is describable and
identifiable by a IVOA Identifier; while it is technically possible to
publish such records, the authors of such records should probably be
asked to use a more specific type.
- vr:service
- A resource that can be invoked by a client to perform some action on
its behalf
- vs:catalogservice
- A service that interacts with one or more specified tables having
some coverage of the sky, time, and/or frequency.
- vs:dataservice
- A service for accessing astronomical data; publishers choosing
this over
vs:catalogservice probably intend to communicate
that there are no actual sky positions in the tables exposed.
- vs:datacollection
- A schema as a logical grouping of data which, in general, is
composed of one or more accessible datasets. Use the
rr.relationship table to find out services that allow
access to the data (the served_by relation), and/or look for values for
/accessURL in
rr.res_detail.
- vstd:standard
- A description of a standard specification.
The status attribute of vr:Resource is
considered an implementation detail of the XML serialization and is not
kept here. Neither inactive nor deleted
records may be kept in the resource table. Since all
other tables in the relational registry should keep a foreign key on the
ivoid column, this implies that only metadata on
active records
is being kept in the relational registry. In other words, users can
expect a resource to exist and work if they find it in a relational
registry .
Name
Utype | Type | Description |
|---|
ivoid
xpath:identifier | adql:VARCHAR(*) | Unambiguous reference to the resource conforming to the IVOA standard for identifiers |
res_type
xpath:@xsi:type | adql:VARCHAR(*) | Resource type (something like vs:datacollection, vs:catalogservice, etc) |
created
xpath:@created | adql:TIMESTAMP | The UTC date and time this resource metadata description was created. This timestamp must not be in the future. This time is not required to be accurate; it should be at least accurate to the day. Any insignificant time fields should be set to zero. |
short_name
xpath:shortName | adql:VARCHAR(*) | A short name or abbreviation given to something. This name will be used where brief annotations for the resource name are required. Applications may use to refer to this resource in a compact display. One word or a few letters is recommended. No more than sixteen characters are allowed. |
res_title
xpath:title | adql:VARCHAR(*) | The full name given to the resource |
updated
xpath:@updated | adql:TIMESTAMP | The UTC date this resource metadata description was last updated. This timestamp must not be in the future. This time is not required to be accurate; it should be at least accurate to the day. Any insignificant time fields should be set to zero. |
content_level
xpath:content/contentLevel | adql:VARCHAR(*) | Description of the content level or intended audience |
res_description
xpath:content/description | adql:VARCHAR(*) | An account of the nature of the resource. |
reference_url
xpath:content/referenceURL | adql:VARCHAR(*) | URL pointing to a human-readable document describing this resource. |
creator_seq
xpath:curation/creator/name | adql:VARCHAR(*) | The creator(s) of the resource in the order given by the resource record author. |
content_type
xpath:content/type | adql:VARCHAR(*) | Nature or genre of the content of the resource |
source_format
xpath:content/source/@format | adql:VARCHAR(*) | The format of source_value. Recognized values include "bibcode", referring to a standard astronomical bibcode (http://cdsweb.u-strasbg.fr/simbad/refcode.html). |
source_value
xpath:content/source | adql:VARCHAR(*) | A bibliographic reference from which the present resource is derived or extracted. |
res_version
xpath:curation/version | adql:VARCHAR(*) | Label associated with creation or availablilty of a version of a resource. |
region_of_regard
xpath:coverage/regionOfRegard | adql:REAL | A single numeric value representing the angle, given in decimal degrees, by which a positional query against this resource should be "blurred" in order to get an appropriate match. |
waveband
xpath:coverage/waveband | adql:VARCHAR(*) | A hash-separated list of regions of the electro-magnetic spectrum that the resource's spectral coverage overlaps with. |
rights
xpath:rights | adql:VARCHAR(*) | Information about rights held in and over the resource. |
This table should have the ivoid column explicitly set
as its primary key.
The following columns MUST be lowercased during ingestion:
ivoid, res_type, content_level,
content_type, source_format,
waveband.
Clients are advised to query the res_description and
res_title columns
using the the ivo_hasword function, and to use
ivo_hashlist_has on content_level,
content_type,
waveband, and rights.
The row for region_of_regard
TAP_SCHEMA.tables must have deg in its
unit column.
This table subsumes the contact, publisher, contributor,
and creator members of the
VOResource data model. They have been combined into a single table to
reduce the total number of tables, and also in anticipation of a unified
data model for such entities in future versions of VOResource.
The actual role is given in the base_role column, which
can be one of contact, publisher, contributor, or
creator. Depending on this value, here are the xpaths
for the table fields (we have abbreviated
/curation/publisher
as cp, /curation/contact as co, /curation/creator
as cc,
and /curation/contributor as cb):
| base_role value |
contact |
publisher |
creator |
contributor |
|---|
| role_name |
co/name |
cp |
cc/name |
cb |
| role_ivoid |
co/name/@ivo-id |
cp/@ivo-id |
cc/name/@ivo-id |
cb/@ivo-id |
| address |
co/address |
N/A |
N/A |
N/A |
| email |
co/email |
N/A |
N/A |
N/A |
| telephone |
co/telephone |
N/A |
N/A |
N/A |
| logo |
co/logo |
N/A |
cc/logo |
N/A |
Not all columns are available for each role type in VOResource. For
example, contacts have no logo, and creators no telephone members. Unavailable
metadata (marked with N/A in the above table) MUST be represented with NULL
values in the corresponding columns.
Note that, due to current practice in the VO, it is not easy to
predict what role_name will contain; it could be a single
name, where again the actual format is unpredictable
(full first name, initials in front or behind, or
even a project name), but it could as well be a full author list. Thus,
when matching against role_names, you will have to use
rather lenient regular expressions. Changing this, admittedly
regrettable, situation would
probably require a change in the VOResource schema.
Name
Utype | Type | Description |
|---|
ivoid
xpath:/identifier | adql:VARCHAR(*) | The parent resource. |
role_name
N/A | adql:VARCHAR(*) | The real-world name or title of a person or organization |
role_ivoid
N/A | adql:VARCHAR(*) | An IVOA identifier of a person or organization |
street_address
N/A | adql:VARCHAR(*) | A mailing address for a person or organization |
email
N/A | adql:VARCHAR(*) | An email address the entity can be reached at |
telephone
N/A | adql:VARCHAR(*) | A telephone number the entity can be reached at |
logo
N/A | adql:VARCHAR(*) | URL pointing to a graphical logo, which may be used to help identify the entity |
base_role
N/A | adql:VARCHAR(*) | The role played by this entity; this is one of contact, publisher, and creator |
The ivoid column should be an explicit foreign key into
the resource table. It is recommended to maintain indexes
on at least the role_name column, ideally in a way that
supports regular expressions.
The following columns MUST be lowercased during ingestion:
ivoid, role_ivoid,
base_role.
Clients are advised to query the remaining columns, in particular
role_name,
case-insensitively, e.g., using ivo_nocasematch.
Since subject queries are expected to be frequent and perform relatively
complex checks (e.g., resulting from thesaurus queries in the clients), the
subjects are kept in a separate table rather than being hash-joined like other
string-like 1:n members of resource.
Name
Utype | Type | Description |
|---|
ivoid
xpath:/identifier | adql:VARCHAR(*) | The parent resource. |
res_subject
xpath:subject | adql:VARCHAR(*) | Topics, object types, or other descriptive keywords about the resource. |
The ivoid column should be an explicit foreign key into
resource. It is recommended to index the
res_subject column, preferably in a way that allows to process
case-insensitive and pattern queries using the index.
The ivoid column MUST be lowercased during
ingestion. Clients are advised query the res_subject column
case-insensitively, e.g., using ivo_nocasematch.
The capability table describes a resource's modes of interaction; it only
contains the members of the base type vr:Capability.
Members of derived types are kept in the res_detail table
(see table_res_detail).
The table has an
integer-typed column cap_index to disambiguate multiple
capabilities on a single resource. Operators are free to choose the
actual values as convenient, although it is recommended to just
enumerate the capabilities in their physical sequence per-resource.
Name
Utype | Type | Description |
|---|
ivoid
xpath:/identifier | adql:VARCHAR(*) | The parent resource. |
cap_index
N/A | adql:SMALLINT | Running number of this capability within the resource. |
cap_type
xpath:@xsi:type | adql:VARCHAR(*) | The type of capability covered here. |
cap_description
xpath:description | adql:VARCHAR(*) | A human-readable description of what this capability provides as part of the over-all service |
standard_id
xpath:@standardID | adql:VARCHAR(*) | A URI for a standard this capability conforms to. |
This table should have an explicit primary key made up of
ivoid and cap_index.
The ivoid column should be
an explicit foreign key into resource.
It is recommended to maintain indexes on at least the
cap_type and standard_id columns.
The following columns MUST be lowercased during ingestion:
ivoid, cap_type, standard_id.
Clients are advised to query the cap_description column
using the ivo_hasword function.
The res_schema table corresponds to VODataService's
schema element. It has been renamed to avoid clashes with
the SQL reserved word SCHEMA.
The table has an integer-typed column schema_index
to disambiguate multiple schema elements on a single resource.
Operators are free to choose the actual values as convenient, although
it is recommended to just enumerate the schema elements in their
physical sequence per-resource. The xpath
count(preceding-sibling::schema) gives one possible
set of values for this column; it has not been prescribed via a utype,
though, since other enumeration schemes are valid as well.
Name
Utype | Type | Description |
|---|
ivoid
xpath:/identifier | adql:VARCHAR(*) | The parent resource. |
schema_index
N/A | adql:SMALLINT | A running number for the res_schema rows belonging to a resource. |
schema_description
xpath:description | adql:VARCHAR(*) | A free text description of the tableset explaining in general how all of the tables are related. |
schemaname
xpath:name | adql:VARCHAR(*) | A name for the set of tables. |
schema_title
xpath:title | adql:VARCHAR(*) | A descriptive, human-interpretable name for the table set. |
schema_utype
xpath:utype | adql:VARCHAR(*) | An identifier for a concept in a data model that the data in this schema as a whole represent. |
This table should have an explicit primary key made up of
ivoid and schema_index. The
ivoid column should be an explicit foreign key into
resource.
The following columns MUST be lowercased during ingestion:
ivoid, schema_name, schema_utype.
Clients are advised to query the schema_description
and schema_title columns
using the the ivo_hasword function.
The res_table table models VODataService's
table element. It has been renamed to avoid name clashes
with the SQL reserved word TABLE.
VODataService 1.0 had a similar element that was a direct child of
resource. Ingestors should also accept such tables, as there are still
numerous active VODataService 1.0 resources in the Registry at the time
of writing (this is the reason for the alternative in the table xpath).
The table contains an integer-typed column table_index to
disambiguate multiple tables on a single resource. Operators are free to
choose the actual values as convenient, although it is recommended to just
enumerate the tables in their physical sequence per-resource (but not per
schema—table_index MUST be unique within a resource).
Name
Utype | Type | Description |
|---|
ivoid
xpath:/identifier | adql:VARCHAR(*) | The parent resource. |
schema_index
N/A | adql:SMALLINT | Index of the schema this table belongs to, if it belongs to a schema (otherwise NULL). |
table_description
xpath:description | adql:VARCHAR(*) | A free-text description of the table's contents |
table_name
xpath:name | adql:VARCHAR(*) | The fully qualified name of the table. This name should include all catalog or schema prefixes needed to distinguish it in a query. |
table_index
N/A | adql:INTEGER | An artificial counter for the tables belonging to a resource |
table_title
xpath:title | adql:VARCHAR(*) | A descriptive, human-interpretable name for the table |
table_type
xpath:@type | adql:VARCHAR(*) | A name for the role this table plays. Recognized values include "output", indicating this table is output from a query; "base_table", indicating a table whose records represent the main subjects of its schema; and "view", indicating that the table represents a useful combination or subset of other tables. Other values are allowed. |
table_utype
xpath:utype | adql:VARCHAR(*) | An identifier for a concept in a data model that the data in this table as a whole represent. |
This table should have an explicit primary key made up of
ivoid and table_index. The
ivoid column should be an explicit
foreign key into resource. It is recommended to
maintain an index on at least the table_description
column, ideally one suited for queries with ivo_hasword.
The following columns MUST be lowercased during ingestion:
ivoid, table_name, table_type,
table_utype.
Clients are advised to query the table_description
and table_title columns
using the the ivo_hasword function.
The table_column table models the content of VOResource's
column element. The table has been renamed to avoid
a name clash with the SQL reserved word COLUMN.
Since it is expected that queries for column properties will be
fairly common in advanced queries, it is the column table that has the
unprefixed versions of common member names (name, ucd,
utype, etc).
The flag column contains a concatenation of all values
of a column element's flag children, separated
by hash characters. Use the ivo_hashlist_has function in
queries against flag.
The table_column table also includes information from
VODataService's data type concept. VODataService 1.1 includes several type
systems (VOTable, ADQL, Simple). The
type_system column contains the value of the column's
datatype child, with the VODataService XML prefix fixed
to vs; hence, this column will contain one of NULL,
vs:TAPType,
vs:SimpleDataType, and vs:VOTableType.
Name
Utype | Type | Description |
|---|
ivoid
xpath:/identifier | adql:VARCHAR(*) | The parent resource. |
table_index
N/A | adql:INTEGER | Index of the table this column belongs to. |
name
xpath:name | adql:VARCHAR(*) | The name of the column. |
ucd
xpath:ucd | adql:VARCHAR(*) | A unified content descriptor that describes the scientific content of the parameter. |
unit
xpath:unit | adql:VARCHAR(*) | The unit associated with all values in the column. |
utype
xpath:utype | adql:VARCHAR(*) | An identifier for a role in a data model that the data in this column represents. |
std
xpath:@std | adql:SMALLINT | If 1, the meaning and use of this parameter is reserved and defined by a standard model. If 0, it represents a database-specific parameter that effectively extends beyond the standard. |
datatype
xpath:dataType | adql:VARCHAR(*) | The type of the data contained in the column. |
extended_schema
xpath:dataType/@extendedSchema | adql:VARCHAR(*) | An identifier for the schema that the value given by the extended attribute is drawn from. |
extended_type
xpath:dataType/@extendedType | adql:VARCHAR(*) | A custom type for the values this column contains. |
arraysize
xpath:dataType/@arraysize | adql:VARCHAR(*) | The shape of the array that constitutes the value, e.g., 4, *, 4*, 5x4, or 5x*, as specified by VOTable. |
delim
xpath:dataType/@delim | adql:VARCHAR(*) | The string that is used to delimit elements of an array value when arraysize is not '1'. |
type_system
xpath:dataType/@xsi:type | adql:VARCHAR(*) | The type system used, as a QName with a canonical prefix; this will ususally be one of vs:simpledatatype, vs:votabletype, and vs:taptype. |
flag
xpath:flag | adql:VARCHAR(*) | Hash-separated keywords representing traits of the column. Recognized values include "indexed", "primary", and "nullable". |
column_description
xpath:description | adql:VARCHAR(*) | A free-text description of the column's contents. |
The pair ivoid, table_index should be an
explicit foreign key into res_table. It is recommended to
maintain indexes on at least the column_description,
name, ucd, and utype columns,
where the index on column_description should ideally be able
to handle queries using ivo_hasword.
The following columns MUST be lowercased during ingestion:
ivoid, name, ucd,
utype, datatype, type_system.
Clients are advised to query the description
column using the ivo_hasword function, and to query
the flag column using the ivo_hashlist_has
function.
The interface table subsumes both the
vr:Interface and vr:accessURL types from
VOResource. The integration of accessURL into
the interface table means that an interface in the
relational registry can only have one access URL, where in VOResource it
can have many. In practice, this particular VOResource capability has
not been used by registry record authors. Since access URLs are
probably the item most queried for, it seems warranted to save one
indirection when querying for them.
This specification deprecates the maxOccurs="unbounded"
in the definition of Interface's accessURL
child in the XML schema
http://www.ivoa.net/xml/VOResource/v1.0; in future versions
of VOResource, implementations can expect this to be
maxOccurs="1". Meanwhile, implementation behaviour in the
presence of multiple access URLs in an interface is undefined.
The table contains an
integer-typed column intf_index to disambiguate multiple
interfaces of one resource. Operators are free to choose the
actual values as convenient, although it is recommended to just
enumerate the interfaces in their physical sequence per resource.
The query_type column is a hash-joined list (analoguos
to waveband, etc. in the resource table), as
the XML schema allows listing up to two request methods.
This table only contains interface elements from within capabilities.
Interface elements in StandardsRegExt records are ignored in RegTAP,
and they must not be inserted in this table, since doing so would disturb
the foreign key from interface into capability. In other words,
RegTAP requires every interface to have a parent capability.
Analogous to resource.res_type, the
intf_type column contains type names; VOResource extensions
can define new types here, but at the time of writing, the following
types are mentioned in IVOA-recommended schemata:
- vs:paramHTTP
- A service invoked via an HTTP query with either form-urlencoded or
multipart form-data parameters.
- vr:WebBrowser
- A (form-based) interface intended to be accessed interactively by a
user via a web browser.
- vg:OAIHTTP
- A standard OAI PMH interface using HTTP queries with form-urlencoded
parameters.
- vg:OAISOAP
- A standard OAI PMH interface using a SOAP Web Service
interface.
- vr:WebService
- A Web Service that is describable by a WSDL document.
Name
Utype | Type | Description |
|---|
ivoid
xpath:/identifier | adql:VARCHAR(*) | The parent resource. |
cap_index
N/A | adql:SMALLINT | The index of the parent capability. |
intf_index
N/A | adql:SMALLINT | A running number for the interfaces of a resource. |
intf_type
xpath:@xsi:type | adql:VARCHAR(*) | The type of the interface (vr:WebBrowser, vs:ParamHTTP, etc). |
intf_role
xpath:@role | adql:VARCHAR(*) | An identifier for the role the interface plays in the particular capability. If the value is equal to "std" or begins with "std:", then the interface refers to a standard interface defined by the standard referred to by the capability's standardID attribute. |
std_version
xpath:@version | adql:VARCHAR(*) | The version of a standard interface specification that this interface complies with. When the interface is provided in the context of a Capability element, then the standard being refered to is the one identified by the Capability's standardID element. |
query_type
xpath:queryType | adql:VARCHAR(*) | Hash-joined list of expected HTTP method (GET or POST) supported by the service. |
result_type
xpath:resultType | adql:VARCHAR(*) | The MIME type of a document returned in the HTTP response. |
wsdl_url
xpath:wsdlURL | adql:VARCHAR(*) | The location of the WSDL that describes this Web Service. If NULL, the location can be assumed to be the accessURL with '?wsdl' appended. |
url_use
xpath:accessURL/@use | adql:VARCHAR(*) | A flag indicating whether this should be interpreted as a base URL ('base'), a full URL ('full'), or a URL to a directory that will produce a listing of files ('dir'). |
access_url
xpath:accessURL | adql:VARCHAR(*) | The URL at which the interface is found. |
This table should have the pair ivoid, cap_index
as an explicit foreign key into
capability, and the pair ivoid, and
intf_index as an explicit primary key. Additionally, it
is recommended to maintain an index on at least the
intf_type column.
The following columns MUST be lowercased during ingestion:
ivoid, intf_type, intf_role,
std_version, query_type,
result_type, url_use.
Clients are adviced to query query_type using the the
ivo_hashlist_has function.
The intf_param table keeps information on the parameters
available on interfaces. It is therefore closely related to
table_column, but the differences between the two are
significant enough to warrant a separation between the two tables.
Since the names of common column attributes are used where applicable in
both tables (e.g., name, ucd, etc), the two tables cannot be (naturally)
joined.
Name
Utype | Type | Description |
|---|
ivoid
xpath:/identifier | adql:VARCHAR(*) | The parent resource. |
intf_index
N/A | adql:SMALLINT | The index of the interface this parameter belongs to. |
name
xpath:name | adql:VARCHAR(*) | The name of the parameter. |
ucd
xpath:ucd | adql:VARCHAR(*) | A unified content descriptor that describes the scientific content of the parameter. |
unit
xpath:unit | adql:VARCHAR(*) | The unit associated with all values in the parameter. |
utype
xpath:utype | adql:VARCHAR(*) | An identifier for a role in a data model that the data in this parameter represents. |
std
xpath:@std | adql:SMALLINT | If 1, the meaning and use of this parameter is reserved and defined by a standard model. If 0, it represents a database-specific parameter that effectively extends beyond the standard. |
datatype
xpath:dataType | adql:VARCHAR(*) | The type of the data contained in the parameter. |
extended_schema
xpath:dataType/@extendedSchema | adql:VARCHAR(*) | An identifier for the schema that the value given by the extended attribute is drawn from. |
extended_type
xpath:dataType/@extendedType | adql:VARCHAR(*) | A custom type for the values this parameter contains. |
arraysize
xpath:dataType/@arraysize | adql:VARCHAR(*) | The shape of the array that constitutes the value, e.g., 4, *, 4*, 5x4, or 5x*, as specified by VOTable. |
delim
xpath:dataType/@delim | adql:VARCHAR(*) | The string that is used to delimit elements of an array value when arraysize is not '1'. |
param_use
xpath:@use | adql:VARCHAR(*) | An indication of whether this parameter is required to be provided for the application or service to work properly (one of required, optional, ignored, or NULL). |
param_description
xpath:description | adql:VARCHAR(*) | A free-text description of the column's contents. |
The pair
ivoid, intf_index
should be an explicit foreign key into interface.
The remaining requirements and conventions are as per
section table_table_column
where applicable, and param_description taking the role
of column_description.
The relationship element is a slight denormalization of the
vr:Relationship type: Whereas in VOResource, a single
relationship element can take several IVORNs, in the relational model,
the pairs are stored directly. It is straightforward to translate
between the two representations in the database ingestor.
Name
Utype | Type | Description |
|---|
ivoid
xpath:/identifier | adql:VARCHAR(*) | The parent resource. |
relationship_type
xpath:relationshipType | adql:VARCHAR(*) | The named type of relationship; this can be mirror-of, service-for, served-by, derived-from, related-to, or something user-defined. |
related_id
xpath:relatedResource/@ivo-id | adql:VARCHAR(*) | The URI form of the IVOA identifier for the resource refered to. |
related_name
xpath:relatedResource | adql:VARCHAR(*) | The name of resource that this resource is related to. |
The
ivoid column should be an explicit foreign key into the
resoure table. You should index at least the
related_id column.
The following columns MUST be lowercased during ingestion:
ivoid, relationship_type,
related_id.
The validation subsumes the
vr:validationLevel-typed members of both vr:Resource
and vr:Capability.
If the cap_index column in NULL the
validation comprises the entire resource. Otherwise, only the
referenced capability has been validated.
While it is recommended that harvesters only accept resource records
from their originating registries, it is valuable to gather validation
results from various sources. Hence, harvesters for the relational
registry may choose to obtain validation data from the OAI-PMH endpoints
of various registries by not harvesting just for the ivo_managed set and
generate rr.validation rows from these records. This can
trigger potentially problematic behaviour when the original registry
updates its resource record in that naive implementations will lose all
third-party validation rows; this may actually be the correct behaviour,
since an update of the registry record might very well indicate
validation-relevant changes in the underlying services. Implementations
are free to handle or ignore validation results as they see fit, and
they may add validation results of their own.
For convenicence, we repeat the definition of the validation
levels from std:RM:
- 0
- The resource has a description that is stored in a registry. This
level does not imply a compliant description.
- 1
- In addition to meeting the level 0 definition, the resource
description conforms syntactically to this standard and to the encoding
scheme used.
- 2
- In addition to meeting the level 1 definition, the resource
description refers to an existing resource that has been demonstrated to
be functionally compliant. When the resource is a service, it is
considered to exist and to be functionally compliant if use of the
Service.InterfaceURL or Service.BaseURL responds without error when used
as intended by the resource. If the service is a standard one, it
must also demonstrate the response is syntactically compliant with the
service standard in order to be considered functionally compliant. If
the resource is not a service, then the ReferenceURL must be shown to
return a document without error.
- 3
- In addition to meeting the level 2 definition, the resource
description has been inspected by a human and judged to comply
semantically to this standard as well as meeting any additional minimum
quality criteria (e.g., providing values for important but non-required
metadata) set by the human inspector. [...]
- 4
- In addition to meeting the level 3 definition, the resource
description meets additional quality criteria set by the human inspector
and is therefore considered an excellent description of the resource.
Consequently, the resource is expected to operate well as part of a
VO application or research study.
Name
Utype | Type | Description |
|---|
ivoid
xpath:/identifier | adql:VARCHAR(*) | The parent resource. |
validated_by
xpath:validationLevel/@validatedBy | adql:VARCHAR(*) | The IVOA ID of the registry or organisation that assigned the validation level. |
val_level
xpath:validationLevel | adql:SMALLINT | A numeric grade describing the quality of the resource description, when applicable, to be used to indicate the confidence an end-user can put in the resource as part of a VO application or research study. |
cap_index
N/A | adql:SMALLINT | If non-NULL, the validation only refers to the the capability referenced here. |
The
ivoid should be an explicit foreign key into resource.
Note, however, that ivoid, cap_index is
not a foreign key into capability since
cap_index may be NULL (in case the validation
addresses the entire resource).
The following columns MUST be lowercased during ingestion:
ivoid, validated_by.
The res_date table contains information gathered from
vr:Curation's date children.
Name
Utype | Type | Description |
|---|
ivoid
xpath:/identifier | adql:VARCHAR(*) | The parent resource. |
date_value
xpath:date | adql:TIMESTAMP | A date associated with an event in the life cycle of the resource. |
value_role
xpath:date/@role | adql:VARCHAR(*) | A string indicating what the date refers to, e.g., created, availability, updated. |
The ivoid column should be an explicit foreign key into
resource.
The following columns MUST be lowercased during ingestion:
ivoid, value_role.
The res_detail table is RegTAP's primary means for
extensibility as well as a fallback for less-used simple
metadata. Conceptually, it stores triples of resource entity
references, resource xpaths,
and values, where resource entities can be resource records themselves
or capabilities. Thus, metadata with values that are either
string-valued or sets of strings can be represented in this table.
As long as the metadata that needs to be represented in the
relational registry for new VOResource extensions is simple enough, no changes to the schema defined
here will be neccessary as these are introduced. Instead, the extension itself simply defines
new xpaths to be added in res_detail.
Some complex metadata—tr:languageFeature or
vstd:key being examples—cannot be kept in this table.
If a representation of such information in the relational registry is
required, this standard will need to be changed.
Appendix d_u_list gives a list
of resource xpaths from the registry extensions
that were recommendations at the time of writing.
For the resource xpaths marked with an exclamation mark there,
xpath/value pairs MUST be generated whenever the corresponding
metadata items are given in a resource record.
For the remaining resource xpaths, these pairs should be provided if
convenient; they mostly concern test queries and other curation-type
information that, while unlikely to be useful to normal users, may be
relevant to curation-type clients that, e.g., ascertain the continued
operation of services.
Some detail values must be interpreted case-insensitively; this
concerns, in particular, IVORNs like the TAP data model type. For other
rows—the test queries are immediate examples—, changing the case
will likely break the data. In order to avoid having to give and
implement case normalization rules by detail xpath, no case normalization
is done on detail values at all, and users and clients will have to use
the ivo_nocasematch user defined function (see section
adqludf) when locating
case-insensitive values. For the resource xpaths given in Appendix d_u_list, this concerns all items with xpaths ending
in @ivo-id.
Individual
ingestors
MAY choose to expose additional metadata using other utypes, provided
they are formed according to the rules in
section vorutypes (this rule is intended
to minimize the risk of later clashes).
In addition to the metadata listed in this specification,
metadata defined in future
IVOA-approved VOResource extensions MUST or SHOULD be present in
res_details as the extensions require it.
Name
Utype | Type | Description |
|---|
ivoid
xpath:/identifier | adql:VARCHAR(*) | The parent resource. |
cap_index
N/A | adql:SMALLINT | The index of the parent capability; if NULL the xpath-value pair describes a member of the entire resource. |
detail_xpath
N/A | adql:VARCHAR(*) | The xpath of the data item |
detail_value
N/A | adql:VARCHAR(*) | (Atomic) value of the member |
The ivoid column should be an explicit foreign key into
resource. It is recommended to maintain indexes on
at least the columns
detail_xpath and detail_value, where the
indices on detail_value should ideally work for both direct
comparisons and searches using the ivo_nocasematch
function.
The following column MUST be lowercased during ingestion:
ivoid. Clients are advised to
use the ivo_nocasematch function to search in
detail_value if the values are to be compared
case-insensitively (e.g., all IVORNs).
This section contains sample queries to the relational registry,
mostly contributed as use cases by various members of the IVOA Registry
working group. They are intended as an aid in designing relational
registry queries, in particular for users new to the data model.
Where these queries locate access URLs of standard services, they
contain intf_role='std' conditions. Desktop clients should
probably include this to weed out nonstandard interfaces on standard
capabilities, which are rare but technically legal; on the other hand,
there are fairly standards-compliant interfaces for which the operator
neglected to set role="std", and the respective services
would be missed with such a condition. Hand-crafted queries can probably
typically do without restraining role since most standard capabilities
in real-world resources only have
one—standard—interface.
TAP accessURLs
Problem: Find all TAP services; return their accessURLs
As the capability type is in rr.capability, whereas the access URL can be
found from rr.interface, this requires
a join:
SELECT ivoid, access_url
FROM rr.capability
NATURAL JOIN rr.interface
WHERE standard_id='ivo://ivoa.net/std/tap'
AND intf_role='std'
Other standard_ids relevant here include:
ivo://ivoa.net/std/registry for OAI-PMH services,ivo://ivoa.net/std/sia for SIA services,ivo://ivoa.net/std/conesearch for SCS services,
andivo://ivoa.net/std/ssa for SSA services.
Image Services with Spirals
Problem: Find all SIA services that might have spiral
galaxies
This is somewhat tricky since it is probably hard to image a part
of the sky guaranteed not to have some, possibly distant, spiral galaxy
in it. However, translating the intention into "find all SIA services
that mention spiral in either the subject (from rr.res_subject), the description, or the
title (which are in rr.resource)",
the query would become:
SELECT ivoid, access_url
FROM rr.capability
NATURAL JOIN rr.resource
NATURAL JOIN rr.interface
NATURAL JOIN rr.res_subject
WHERE standard_id='ivo://ivoa.net/std/sia'
AND intf_role='std'
AND (
1=ivo_nocasematch(res_subject, '%spiral%')
OR 1=ivo_hasword(res_description, 'spiral')
OR 1=ivo_hasword(res_title, 'spiral'))
Infrared Image Services
Problem: Find all SIA services that provide infrared
images
The waveband information in rr.resource
comes in hash-separated atoms (which can be
radio, millimeter, infrared, optical, uv, euv, x-ray, or gamma-ray).
For matching those, use the ivo_hashlist_has function as
below. The access URL and the service standard come from rr.interface and rr.capability, respectively.
SELECT ivoid, access_url
FROM rr.capability
NATURAL JOIN rr.resource
NATURAL JOIN rr.interface
WHERE standard_id='ivo://ivoa.net/std/sia'
AND intf_role='std'
AND 1=ivo_hashlist_has('infrared', waveband)
Catalogs with Redshifts
Problem: Find all searchable catalogs (i.e., cone search
services) that provide a column containing redshifts
Metadata on columns exposed by a service are contained in rr.table_column. Again, this table can be
naturally joined with rr.capability
and rr.interface:
SELECT ivoid, access_url
FROM rr.capability
NATURAL JOIN rr.table_column
NATURAL JOIN rr.interface
WHERE standard_id='ivo://ivoa.net/std/conesearch'
AND intf_role='std'
AND ucd='src.redshift'
Sometimes you want to match a whole set of ucds. Frequently the
simple regular expressions of SQL will help, as in
AND ucd LIKE 'pos.parallax%' . When that is not enough,
use boolean OR expressions
Names from an Authority
Problem: Find all the resources published by a certain
authority
An "authority" within the VO is something that hands out identifiers.
You can tell what authority a record came from by looking at the "host
part" of the IVO identifier, most naturally obtained from rr.resource. Since ADQL cannot actually parse
URIs, we make do with simple string matching:
SELECT ivoid
FROM rr.resource
WHERE ivoid LIKE 'ivo://org.gavo.dc%'
Records Published by X
Problem: What registry records are there from a given
publisher?
This uses the rr.res_role table both
match names (in this case, a publisher that has "gavo" in its name) and
to ascertain the named entity actually publishes the resource (rather
than, e.g., just being the contact in case of trouble). The result is a
list of ivoids in this case. You could join this with any other
table in the relational registry to find out more about these
services.
SELECT ivoid
FROM rr.res_role
WHERE 1=ivo_nocasematch(role_name, '%gavo%')
AND base_role='publisher'
or, if the publisher actually gives its ivo-id in the registry
records,
SELECT ivoid
FROM rr.res_role
WHERE role_ivoid='ivo://ned.ipac/ned'
AND base_role='publisher'
Records from Registry
Problem: What registry records are
there originating from registry X?
This is mainly a query interesting for registry maintainers. Sill,
it is a nice example for joining with the
rr.res_detail table, in this case to
first get a list of all authorities managed by the CDS registry.
SELECT ivoid FROM rr.resource
RIGHT OUTER JOIN (
SELECT 'ivo://' || detail_value || '%' AS pat FROM rr.res_detail
WHERE detail_xpath='/managedAuthority'
AND ivoid='ivo://cds.vizier/registry')
AS authpatterns
ON (resource.ivoid LIKE authpatterns.pat)
Locate RegTAP services
Problem: Find all TAP endpoints offering the relational
registry
This is the discovery query for RegTAP services themselves; note how
this combines rr.res_detail pairs with
rr.capability and rr.interface to locate the desired protocol
endpoints.
SELECT access_url
FROM rr.interface
NATURAL JOIN rr.capability
NATURAL JOIN rr.res_detail
WHERE standard_id='ivo://ivoa.net/std/tap'
AND intf_role='std'
AND detail_xpath='/capability/dataModel/@ivo-id'
AND 1=ivo_nocasematch(detail_value, 'ivo://ivoa.net/std/regtap/vor')
TAP with Physics
Problem: Find all TAP services
exposing a table with certain features
"Certain features" could be "have some word in their description
and having a column with a certain UCD". Either way, this kind of query
fairly invariably combines the usual rr.capability and rr.interface for service location with
rr.table_column for the column metadata
and rr.res_table for properties of tables.
SELECT ivoid, access_url, name, ucd, column_description
FROM rr.capability
NATURAL JOIN rr.interface
NATURAL JOIN rr.table_column
NATURAL JOIN rr.res_table
WHERE standard_id='ivo://ivoa.net/std/tap'
AND intf_role='std'
AND 1=ivo_hasword(table_description, 'quasar')
AND ucd='phot.mag;em.opt.v'
Theoretical SSA
Problem: Find all SSAP services that
provide theoretical spectra
The metadata required to solve this problem is found in the SSAP
registry extension and is thus kept in rr.res_detail:
SELECT access_url
FROM rr.res_detail
NATURAL JOIN rr.capability
NATURAL JOIN rr.interface
WHERE detail_xpath='/capability/dataSource'
AND intf_role='std'
AND standard_id='ivo://ivoa.net/std/ssa'
AND detail_value='theory'
Find Contact Persons
Problem: The service at
http://dc.zah.uni-heidelberg.de/__system__/tap/run/tap is down, who can
fix it?
This uses the rr.res_role table and
returns all information on it based on the IVORN of a service that in
turn was obtained from rr.interface. You
could restrict to the actual technical contact person by requiring
base_role='contact'.
SELECT DISTINCT base_role, role_name, email
FROM rr.res_role
NATURAL JOIN rr.interface
WHERE access_url='http://dc.zah.uni-heidelberg.de/__system__/tap/run/tap'
Related Capabilities
Problem: Get the capabilities of all services serving a
specific resource (typically, a data collection)
In the VO, data providers can register pure data collections without
access options (or just furnished with a link to a download). They can
then declare that their data can be "served-by" some other resource,
typically a TAP service or some collective service for a number of
instruments. To locate the access options to the data itself, inspect
rr.relationship and use it to select records
from rr.capability.
SELECT *
FROM rr.relationship AS a
JOIN rr.capability AS b
ON (a.related_id=b.ivoid)
WHERE relationship_type='served-by'
