This document presents a model for describing certain types of numerical computer simulations and certain types of simulation post-processing products. The model was originally envisioned to be used in the query part of the Simple Numerical Access Protocol (SNAP), and in discovery of interesting SNAP services in the first place. After investigating the application domain carefully, and having tried to develop and implement a preliminary version of the former Simple Numerical Access Protocol, ADD A REFERENCE TO THE LAST SNAP DOCUMENT? RV: In modelling (data, or physics or whatever) a given approach is never better per se (otherwize everybody should have done it in the past), in particular when the field under study is not unconnected to other related modelling efforts in close fields. This is the case of modelling in IVOA (not VO!) context. The community will be convinced it is better because their promoters are also able to demonstrate that other approaches - those the community decided to promote as a paradigm - failed to solve some major issues. We thus have to demonstrate somewhere in the document that our previous Simple(NAP) attempt failed because of some serious lacks. This needs a critical implementation analysis on a few test-beds (Laurent? that's what you did on GalMer and the first GalICS schema I guess). we have decided to leave the concept of designing a DAL-like SxAP protocol for simulations. Instead we have split up the effort into two separate efforts that can be used each in their own right, though their is a clear link between them. This document discusses the firsts of these, which we have named the Simulation Database, and will have the acronym SimDB. The second will be developed further in a separate effort and is called the Simulation Data Access Protocol (SimDAP, "Sim" stands for "Simulation", not "Simple"!).

Following SNAP, SimDB only explicitly considers simulations for systems that represent a space-time sub-volume of the universe and (part of) its material contents. Examples of such simulations are cosmological, pure dark matter N-body simulations of the large-scale structure of the universe; adaptive mesh refinement (AMR) simulations following the evolution of a galaxy cluster using full hydrodynamics; a simulation of the evolution of a globular cluster using a combination of tools, together simulating the various types of physics @@ TODO reference to MODEST-like activities; or simulations calculating the few seconds of a supernova explosion in full 3D.

In general these simulations will evolve this system forward in time and are able to produce snapshots, representing the state of the system, a 3D volume of space, at a number of discrete times (though there are alternatives: light cone simulations, individual particle orbits). These direct, raw results of simulations we call Level-0 products, following similar terminology for observations. SimDB also covers Level-1 products, which consist of the results of certain types of post-processing of simulations, namely those products that in some form create an alternative representation of a spatial sub-volume of the universe. For example a density field calculated on a regular grid, derived created from an N-body or an AMR simulation; a cluster catalogue derived using some group finder applied to a cosmological simulation, or a synthetic galaxy catalogue derived from the cluster catalogue using halo occupation distribution models (HODs) or semi-analytical models (SAMs).

We do not make any restrictions on the type @@ TODO DEFINE WHAT IS A 'TYPE'? (ASTRO-PHYSICAL OBJECT?) @@ of systems being simulated, or the size of the simulation, or the way the system is represented in the simulation code and results. We also make no restrictions on the type of "observables" produced by the simulations.

The SimDAP specification will includes protocols for services that process Level-0 or Level-1 results and produce other Level-1 results. The allowed services deal with selecting the results in a sub-volume of the complete result, sampling a regular 3-dimensional grid, etc. SimDAP also allows for services, that do not produce SimDB-like, Level-0 or 1 products. Examples are projections, 1D or 2D samplings. But also custom services will be allowed, for example calculating statistical properties such as correlation functions or power spectra in cosmological simulations. A more detailed description of SimDAP is outside of the main scope of this note.

SimDB is a specification that defines the interface to a database containing meta data describing simulations. To this end it contains two main parts, one is a model for the meta data, the other a protocol for interacting with the database. The model is the core of the specification. It describes the structure of individual data products in the database. We have chosen UML as modelling language, as prescribed by the data modelling working group in the interoperability meeting in Cambridge, UK, May 2003.

The UML model is a logical model (see [..] @@ TODO add reference @@) and forms the basis for physical representations of the data products in the standard language that the IVOA has chosen for such purposes, XML. We derive an XML schema defining valid XML documents directly from the logical model. The SimDB interface will include functions for insetting SimDB data products using such documents, and for retrieving individual, identified data products.

The logical model also forms the basis for a physical representation supporting formulation of queries. For various reasons explained below we have chosen ADQL to be the query language and accordingly we derive from the model a relational schema that defines the tables and columns that can be used in ADQL queries sent to a SimDB implementation. The result of ADQL queries is supposed to be a VOTable, and this will in general not represent a complete SimDB data product. However it can be used to browse the database, finally identifying resources and possibly requesting these from the SimDB as XML documents.

We make very limited assumptions on how a data product discovered in a SimDB can actually be accessed. We only assume there is a web-based service available, identified by a base URL and tagged with a service type. The range of service types will be defined by SimDAP, but it will at least include "download" and "custom". The data model contains an explicit element for indicating which services are available for a given data product, and users may, if they wish, retrieve this information through ADQL queries and follow the links directly. SimDB implementations can and likely will eventually provide SimDAP related functionality, but this is not part of this specification.

It must be possible to find SimDB instances in an IVOA Resource Registry @@TODO add references&&. This implies we need a corresponding resource type, and we have to design its structure. We also assume that one may define resources in the sense of [...] @@ TODO add reference to Resource data model document @@ from within the contents of a SimDB. We take this into account explicitly in the model. The SimDB will have a "getIVOAResource" function, which will execute the appropriate transformation from the internal representation of the SimDB data products to the Resource model's XML representation [...] @@ TODO link to Resource XML schema document@@. This will likely put more requirements on the Registry model itself, maybe requiring extensions to its schema. Possibly a SimDB itself can be an extension registry. This we think can be postponed to a future version of the specification.

We

SimDB defines a common data model for simulations. Following the good practice for database design initiated in [] @@TODO add references&&, we here provide a number of scientific questions one might want to ask such a database. The data model and associated data access protocol need to be sufficiently rich that they can support such questions.

• Scientific goal: investigate baryon wiggles in the evolved density field
  Query: Return all cosmological, pure dark matter, N-body simulations with WMAP 3 initial conditions and a box size of at least 1000 Mpc comoving, containing snapshots at about 10 redshifts between 3 and 0.
• Scientific goal: investigate whether observed structures in X-ray cluster that seem to indicate turbulence, can truly be that.
  Query: return all hydro-dynamical simulations of galaxy clusters of mass at least 10¹⁴ M_sun, that have a model for viscosity included in the simulation. Moreover, return only those simulations that have associated to them an online visualisation service that can produce projected temperature and pressure maps.
• Scientific goal: interpret the possible histories of an observed galaxy merger to calculate possible star formation episodes and compare these to the observed stellar populations.
  Query: Return all simulations of galaxy mergers where the component galaxies have a particular mass ratio and where there are enough snapshots to follow the evolution over a few Gyr.
• Scientific goal: compare the luminosity function of galaxies in the SDSS survey with those in synthetic catalogues.
  Query: Select all cosmological simulations that have produced as secondary product synthetic galaxy catalogues on a light-cone and provide those via an SQL (ADQL?) query interface.
• Scientific goal: compare a number of properties of semi-analytical galaxies in large-scale dark matter simulations predicted by two different set of parameters of a given semi-analytical modelling tool.
  Query: Return the catalogues of semi-analytical galaxies for a select number of semi-analytical runs made with different input parameters in a given large-scale dark matter simulation. @@ TODO MORE PRECISE QUERY @@
• Scientific goal: find N-body and hydrodynamical simulations with comparable rotation curves than a set of spectroscopic observations.
  Query: TODO QUERY
• ...

In the design of the model it is useful to think about the steps a user might go through when querying a database system in various "drilling down" steps. For example the following questions might be asked :

• What system/object is being simulated?
• What physical processes are included?
• How is the system being represented in the simulation (particles (Langrangian), (adaptive) mesh (Eulerian)), both, other?
• Per process:
  • How are the physical processes implemented ?
  • Characterise the numerical approximations (.e.g. resolution, softening parameter)
• What observables are available for the system/object, possibly as function of time? As it is a spatial system, at least simulation boxsize, center-of-mass position.
• What observables are available for the constituents, i.e. what is the schema of the atomic objects?
• Per snapshot, per atomic WHAT AN ATOMIC OBJECT TYPE ?? object type, per variable:
  • Characterise the possible values
  • Characterise the result
• Are post-processing results available?
• Are services/applications available working on the results?
• Which code ran the simulation?
  • Which version of the code ?
• Who did make the simulations?
• What were values of physical parameters?
• How were initial conditions created, what parameters?

• fill a relational database with tables and views representing the SimDB data model from DDL scripts generated from the UML
• create a web-based service that accept XML documents for inserting new simulation results and translates these, using generated code with JAXB annotations, to in memory Java objects
• flush these objects to a relational database using the Java Persistence Architecture (JPA) implementation, structured using the JPA annotations generated on the Java classes. It should be not too hard to support other languages as well if they provide similar simple XML binding and OR-mapping capabilities. Python+Django and C#+LINQ or NHibernate come to mind. @@ TODO check with people knowing more about these technologies @@
• accept ADQL queries that are translated to the appropriate vendor specific SQL (using modules defined by the ADQL effort?) and return a VOTable
• accept requests for identified SimDB resources (using an IVO or implementation specific identifier), translate this into a JPA query to retrieve the object form the database, which is translated to the appropriate XML using the JAXB layer and sent back to the user.

We have tried to find a balance in the level of normalisation of the data model.

Here we discuss the actual contents of the model, though the detailed descritpion

At the root of the SimDB data model is an abstract class called Resource, in the rest of this document we will refere to this as SimDB/Resource. It represents the different types of highest level meta-data objects to be stored in a SimDB. Examples of this are represented as subclasses. First Experiment (SimDB/Experiment), which represents different types of experiments that have been performed (run/executed/...) and have produced the results that SimDB users may be interested in. Examples of SimDB/Experiment-s are first simulations, but also the various post-processing operations transforming simulation results into other products such as halo catalogues, density fields etc.

One of the main differences between the SimDB data model and other data models in the IVOA so far, is that we do not know in advance what types of results we can expect. imho, this difference is the most important and justifies a different approach; should be highlight at the beginning of the note The Spectrum data model describes spectra, the characterisation data model characterises observational results, the model implicit (for now) in SIA deals with 2D images etc. This implies that many features describing these results can be explicitly modeled: Spectra have been taken of sources on the sky, during a certain time persiod, covering a certain wavelength range. And fluxes were measured (in some form). This makes it possible for space/time/wavelength/flux to make explicit appearances in the corresponding models.

In contrast, simulations come in a great variety of types, even if we constrain ourselves to the "3+1D" kind. We can not make many assumptions on the type of objects making up the result, or on the "observables" of these objects. It therefore becomes necessary to add components to the model that allow publishers to describe these explicitly. We do so in the ObjectType hierarchy.

• Should we add units to the properties and not to the characterisations; similar for InputParameter and ParameterSetting

The first question most people want to know about a simulations is: "what is being simulated?". The answer should correspond to a real (astronomical) object, or collection of objects, or possibly a physical process. For SimDB to answer such questions implies that publishers must be able to describe these concepts in the model. We have introduced the TargetObjectType and TargetProcess classes for this.... @@ TODO expand @@.

There are many instances in the data model where we need to describe elements of the SimDB/Resource-s explicitly, because we do not have implicit information based on the context. Examples are the various properties of object types, the target objects and processes etc. Apart from a name and a description we then frequently add an attribute which is supposed to "label" the element according to an assumed standard list of terms. We model this using the

<<ontologyterm>>

• We need to list the onotlogies that we create first attempts on some that do not yet exists.
• We need to have proper locations of machine readable vocabularies
• We need to get feedback on what kind of ontologies we want. Narrower/broader types, fully linked ontologies? etc

The current (May 2008 @@ TODO update when necessary @@) version of the model allows publishers to specify numerical quantities using a real value and a unit. I.e. we do not prescribe units for particular quantities. Allowing this flexibility in units assignment does pose a problem for a query interface that allows user to query on characterisation values and other numerical quantities. ADQL does not include units for example, but a user can not assume that every publisher will use the same unit for for example the typical size of a simulation box. This is even worse of course for the characterisation values of properties that have to be defined in the model and can have any kind of assumed unit.

The goal of the SimDB specification is to define a protocol for querying interesting simulations and related SimDB/Resource-s. Once these have been identified the user should be able to access these simulations. We assume that web services are the means to do so, and allow publishers to indicate such web services as are available for a given Experiment. We assume for now that we know little of the web service beyond some generic types: download, cut-out, extraction, projection, custom. The SimDAP specification is being developed to address those aspects in detail. We assume that there will be a base-URL implementing some standard DAL (VOSI?) like services and leave it up to SimDB-client implementations to interact with these services in standard manners. Only custom services can be directly accessed, and for now many services will necessarily be custom.

• How do we get the complete list of service types? Predefined (as enumeration) in model?

Here we describe how we create physical models out of the logical model. A physical model is (see @@TODO reference to some standard reference on data modelling@@) a representation of the logical model that is adapted to a particular software environment. We present physical representations for the following contexts:

• object types are mapped to tables, one table per object type
• Inheritance hierarchies: JOINED strategy as defined in JPA, i.e. each table only has columns for the attributes and references defined on the corresponding type. Also an ID column that is a PK and also a FK to the ID of the base class' table. Possibly a container column (see below)
• Primary key column: ID NUMERIC(18)
• Foreign key to container: containerId
  plus foreign key and index declaration
• References: <referenceName>Id
  plus foreign key and index declaration.
• Using topological sort of object types based on (extends|container|reference) relations we generated create table statements and ther indexes and foreign keys in blocks. drop table statements in opposite order.
• For each class we create a view named "v_<class name>"
  returns all columns for that class; uses join to base class's view.
• generate a discriminator column on table for root in inheritance hierarchy, stores name of class (must be unique in inheritance hierarchy!)
• attributes mapped to single column if their type is simple (i.e. primitive, or enumeration)
• if attribute's type is dataType mapped to as many columns as the dataType has attributes, with column names the name of the dataType's attributes, prefixed by <attribute-name>_
• For PK columns we use the

The DM WG has mandated (IVOA interoperability meeting, Cambridge, UK, May 2003) that each data model should come with an XML schema that represents valid XML serialisations of the data model. We foresee that this representation can be used to communicate instances of SimDB/Resource-s as XML documents. Such communication can be for registering new SimDB/Resources in a SimDB, or used in message to communicate instances of the SimDB Resource type. Here we shortly describe some of the rules for deriving an XML schema from our logical model.

• object and data types are mapped to comlexType. Object types inherit from a base class taht defines features dealing with identity.
• primitiveType-s are mapped to appropriate simpleType-s
• enumerations are mapped to simpleType-s which are a restriction of xsd:string and have an enumeration element for each literal.
• packages are mapped to namespaces and eahc package has its own file, with dependencies translated to schema imports.
• A root element is generated for each concrete (i.e. non-abstract) root (i.e. not contained in other types) object type.
• attributes are mapped to elements (not attributes!) of the appropriate type.
• collections are mapped to elements of the appropriate type, contained within the complexTYpe of the containing complexType
• references are mapped to a elements of a special purpose base complexType, Reference. The precise definition of this type is postponed until the issues about identity and referencing is resolved. For now it has multiple sub-elements reflecting the different possible ways to refer to other elements.

It is generally the case that contents of databases may be represented in ways that do not conform to one of the standard serialisations. Nothing prevents services to be developed on top of SimDB that represent SimDB/Resource-s or even fragments of these in another form. The standard example would be to have VOTables storing the results of a generic ADQL query of the SimDB/RDB representation. VOTable first introduced the option to have a UTYPE attribute in FIELD definition tags store a pointer to an element in a data model that the column represents.

The previous chapter has defined a number of physical representations of the logical simulation data model. Using these we can implement a database that can store instances of SimDB/Resources. This could be done using an XML database, or using a relational database management system such as Postgres, MySQL or any of the commercial versions. The data model is rather complex, and more hierarchical than most other data models so far defined in the IVOA. Querying such a data model requires a rich query language and we propose to use ADQL working on the relational representation. ADQL produces tabular results, whose structure is completely governed by the query itself. We also assume it possible, once appropriate information is available, to retrieve complete SimDB/Resource-s as XML documents and propose a simple REST-like query interface for that. Such an XML based interface will likely also be used to upload new resources to SimDB implementations taht support that functionality.

We expect no problems in formulating ADQL queries based on the relational representation of the data model described in the previous chapter. We need to require an appropriate protocol for sending these queries to a SimDB service though. In DAL work has started on the Table Access Protocol (TAP) and clearly some version of that seems to be applicable to our situation. However there are some simplifying features. Foremost is that we pre-define the relational schema, so a generic TAP "getMetadata" service seems not necessary. There are likely going to be other standard DAL service features that we need to support (getCapabilities?), but as meta data databases are expected to be relatively small we may again not require the full richness of asynchronous querying, staging, VOSpace and what not.

• (How) does TAP deal with units?
• In TAP, does a table column containing values always have a single UCD and a single Unit?
• Is TAP suited for this kind of meta data databases?

Under this heading we mean a protocol whereby data products can be retrieved through HTTP GET requests. Possibly also they can be POST-ed, or PUT. This needs to be discussed further, but maybe can be punted until a future release. The GET will always only be able to get a complete SimDB resource, serialised to SimDB/XML, similar to the IVOA Resource Registry interface @@ TODO is this actually a correct statement?@@.

Assigning meta-data to describe simulations etc is quite a lot of work if this is to be done aftre the fact. It seems more fruitful to see if simulation codes could make the production of the appropriate documents part of their pipe-line. It is our goal to contact the writers of some of the major simulation packages and see whether they are willing to do so. First contacts with Volker Springel (Gadget), the group in San Diego (Enzo) give us hope that this could be achieved. The TIG should see it as its task to contact more authors of such codes and promote this idea further.

Together with SimDB implementations we need to urge scientists to develop online services for accessing their published simulations. Until the SimDAP specification is further developed these can be custom services, but it is important that services are available asap. This outreach is a task for the TIG.