revision 4595 by michele.sanguillon, Wed Nov 29 16:36:28 2017 UTC revision 4596 by mnullmei, Thu Nov 30 12:48:00 2017 UTC
# Line 51  Line 51
51  \affil{$^1$Laboratoire Univers et Particules de Montpellier, Universit\'{e} de Montpellier, CNRS/IN2P3, France; \email{Michele.Sanguillon@umontpellier.fr}}  \affil{$^1$Laboratoire Univers et Particules de Montpellier, Universit\'{e} de Montpellier, CNRS/IN2P3, France; \email{Michele.Sanguillon@umontpellier.fr}}
52  \affil{$^2$Centre de Donn\'{e}es astronomiques de Strasbourg, Observatoire Astronomique de Strasbourg, Universit\'{e} de Strasbourg, CNRS, Strasbourg, France}  \affil{$^2$Centre de Donn\'{e}es astronomiques de Strasbourg, Observatoire Astronomique de Strasbourg, Universit\'{e} de Strasbourg, CNRS, Strasbourg, France}
53  \affil{$^3$ICube Laboratory, Universit\'{e} de Strasbourg, CNRS, Strasbourg, France}  \affil{$^3$ICube Laboratory, Universit\'{e} de Strasbourg, CNRS, Strasbourg, France}
54  \affil{$^4$ARI/ZAH, Heidelberg, Germany}  \affil{$^4$Zentrum f{\"u}r Astronomie der Universit{\"a}t Heidelberg, Astronomisches Rechen-Institut, Heidelberg, Germany}
55  \affil{$^5$Leibniz Institute for Astrophysics Potsdam, Germany}  \affil{$^5$Leibniz Institute for Astrophysics Potsdam, Germany}
56  \affil{$^6$Laboratoire Univers et Th\'{e}ories, Observatoire de Paris, PSL Research University, CNRS, 92190 Meudon, France}}  \affil{$^6$Laboratoire Univers et Th\'{e}ories, Observatoire de Paris, PSL Research University, CNRS, 92190 Meudon, France}}
57
# Line 59  Line 59
59  % \paperauthor{Sample~Author1}{Author1Email@email.edu}{ORCID_Or_Blank}{Author1 Institution}{Author1 Department}{City}{State/Province}{Postal Code}{Country}  % \paperauthor{Sample~Author1}{Author1Email@email.edu}{ORCID_Or_Blank}{Author1 Institution}{Author1 Department}{City}{State/Province}{Postal Code}{Country}
60
61  \paperauthor{Mich\{e}le Sanguillon}{@email.edu}{0000-0003-0196-6301}{CNRS, Universit\'{e} de Montpellier}{LUPM}{Montpellier}{ }{34095}{France}  \paperauthor{Mich\{e}le Sanguillon}{@email.edu}{0000-0003-0196-6301}{CNRS, Universit\'{e} de Montpellier}{LUPM}{Montpellier}{ }{34095}{France}
62  \paperauthor{Fran\c{c}ois Bonnarel}{francois.bonnarel@astro.unistra.fr}{ }{Universit\'{e} de Strasbourg, CNRS}{Observatoire astronomique de Strasbourg - UMR7550}{Strabourg}{}{67000}{France}  \paperauthor{Fran\c{c}ois Bonnarel}{francois.bonnarel@astro.unistra.fr}{ }{Universit\'{e} de Strasbourg, CNRS}{Observatoire astronomique de Strasbourg - UMR7550}{Strasbourg}{}{67000}{France}
63  \paperauthor{Mireille Louys}{mireille.louys@unistra.fr}{0000-0002-4334-1142}{Universit\'{e} de Strasbourg, CNRS} {ICube Laboratory - UMR7357}{Strabourg}{}{67000}{France}  \paperauthor{Mireille Louys}{mireille.louys@unistra.fr}{0000-0002-4334-1142}{Universit\'{e} de Strasbourg, CNRS} {ICube Laboratory - UMR7357}{Strasbourg}{}{67000}{France}
64  \paperauthor{Markus Nullmeier}{mnullmei@ari.uni-heidelberg.de}{ }{ }{ }{Heidelberg}{State/Province}{Postal Code}{Germany}  \paperauthor{Markus Nullmeier}{mnullmei@ari.uni-heidelberg.de}{ }{ }{ }{Heidelberg}{State/Province}{Postal Code}{Germany}
65  \paperauthor{Kristin Riebe}{kriebe@aip.de}{ }{ }{ }{Potsdam}{State/Province}{14482}{Germany}  \paperauthor{Kristin Riebe}{kriebe@aip.de}{ }{ }{ }{Potsdam}{State/Province}{14482}{Germany}
66  \paperauthor{Mathieu Servillat}{mathieu.servillat@obspm.fr}{0000-0001-5443-4128}{Observatoire de Paris, PSL Research University, CNRS}{LUTH}{Meudon}{}{92195}{France}  \paperauthor{Mathieu Servillat}{mathieu.servillat@obspm.fr}{0000-0001-5443-4128}{Observatoire de Paris, PSL Research University, CNRS}{LUTH}{Meudon}{}{92195}{France}
# Line 71  Line 71
71  % (e.g. an image, spectrum, catalog or single points in a spectral energy distribution diagram or a light curve),  % (e.g. an image, spectrum, catalog or single points in a spectral energy distribution diagram or a light curve),
72  % learn about the people and organizations involved in a project and assess the quality of the dataset as well as the usefulness of the dataset for her own scientific work.  % learn about the people and organizations involved in a project and assess the quality of the dataset as well as the usefulness of the dataset for her own scientific work.
73
74   As presented in a talk at last year's ADASS, the International Virtual Observatory Alliance (IVOA) is working on the definition of a provenance data model,   As presented in a talk at last year's ADASS, the International Virtual Observatory Alliance (IVOA) is working on the definition of a Provenance Data Model,
75   compatible with the W3C PROV model, which shall describe how provenance metadata can be modeled, stored and exchanged in astronomy.   compatible with the W3C PROV model, which shall describe how provenance metadata can be modeled, stored and exchanged in astronomy.
76  % In this poster, we are presenting an overview on the progress of developing (mainly open source) libraries  % In this poster, we are presenting an overview on the progress of developing (mainly open source) libraries
77  % and tools which implement the IVOA provenance data model in order to produce, serve, load and visualize provenance information.  % and tools which implement the IVOA Provenance Data Model in order to produce, serve, load and visualize provenance information.
78  % These implementations are also needed to validate and adjust the data model and the standard definitions for accessing provenance.  % These implementations are also needed to validate and adjust the data model and the standard definitions for accessing provenance.
79  % The implementations include the voprov python package implementing the provenance classes and different serialization formats,  % The implementations include the voprov python package implementing the provenance classes and different serialization formats,
80  % a django web application with different interfaces to access provenance metadata,  % a django web application with different interfaces to access provenance metadata,
81  % provenance developments in the context of CTA  % provenance developments in the context of CTA
82  % and recent developments of a CDS Provenance service, which was tested using RGB compositing and HiPS creation as example use cases.  % and recent developments of a CDS Provenance service, which was tested using RGB compositing and HiPS creation as example use cases.
83
84  In this poster, we present the current status of our developments of libraries and tools, mainly open source, which implement the IVOA Provenance data model in order to  In this poster, we present the current status of our developments of libraries and tools, mainly open source, which implement the IVOA Provenance Data Model in order to
85  produce, serve, load and visualize provenance information. These implementations are also needed to validate and adjust the data model and the standard  produce, serve, load and visualize provenance information. These implementations are also needed to validate and adjust the data model and the standard
86  definitions for accessing provenance. The Provenance tools developed and created for the W3C framework are reused and extended when possible to  definitions for accessing provenance. The provenance tools developed and created for the W3C framework are reused and extended when possible to
87  tackle the domain of astronomical data.  tackle the domain of astronomical data.
88  \end{abstract}  \end{abstract}
89
# Line 97  Line 97
97  \section{Introduction}  \section{Introduction}
98  The International Virtual Observatory Alliance\footnote{\url{http://www.ivoa.net/}}  The International Virtual Observatory Alliance\footnote{\url{http://www.ivoa.net/}}
99  has developed several data models to foster interoperability  has developed several data models to foster interoperability
100  between the various astronomy projects. Even though a lot of objects (spectra, images, simulations, etc.) are already well described,  between diverse astronomy projects. Even though a lot of objects (spectra, images, simulations, etc.) are already well described,
101  some information about how the datasets were produced is still missing.  some parts of the information about how datasets have been produced is still missing.
102
103  That is why the IVOA Data Model Working Group investigates how to model  That is why the IVOA Data Model Working Group investigates how to model
104  provenance information of a dataset, how this information can be stored and how it can be exchanged.  provenance information of a dataset, how this information can be stored and how it can be exchanged.
105  In order to check the validity of the defined model, the group implements the IVOA Provenance Data Model in four environments:  In order to check the validity of the defined model, the group implemented the IVOA Provenance Data Model in four environments:
106  Pollux, CTA, RAVE and at CDS.  Pollux, CTA, RAVE, and one at CDS.
107
108  We present here the tools developed to implement this model in these different contexts.  Here, we present the tools developed to implement this model in these different contexts.
109
110  \section{IVOA Provenance Data Model}  \section{IVOA Provenance Data Model}
111  The IVOA Provenance Data Model  The IVOA Provenance Data Model
112  %\footnote{\url{http://www.ivoa.net/documents/ProvenanceDM/}}  %\footnote{\url{http://www.ivoa.net/documents/ProvenanceDM/}}
113  \citep{IVOAProvenanceDM} follows the W3C Provenance definition  \citep{IVOAProvenanceDM} follows the W3C Provenance definition,
114  i.e. that provenance is information about entities, activities, and people involved in  i.~e., that provenance is information about entities, activities, and people involved in
115  producing a piece of data or thing, which can be used to form assessments about its quality, reliability or trustworthiness''.  producing a piece of data or thing, which can be used to form assessments about its quality, reliability or trustworthiness''.
116
117  The main core classes (\emph{Entity}, \emph{Activity}, \emph{Agent}) and its relations (\emph{wasGeneratedBy}, etc.) have the same name as in the W3C Provenance Data Model \citep{std:W3CProvDM}.  The main core classes (\emph{Entity}, \emph{Activity}, \emph{Agent}) and its relations (\emph{wasGeneratedBy}, etc.) have the same name as in the W3C Provenance Data Model \citep{std:W3CProvDM}.
# Line 133  Line 133
133  % in the following graphic formats: PDF, PNG, SVG.  % in the following graphic formats: PDF, PNG, SVG.
134
135  The voprov package implements the serialization of the IVOA Provenance Data Model.  The voprov package implements the serialization of the IVOA Provenance Data Model.
136  As this model is very close to the W3C one, the voprov library uses the prov  As this model is very close to the W3C one, the voprov library uses the following facilites from prov:
137  developments: PROV-N, PROV-JSON, PROV-XML serialization formats and PDF, PNG, SVG graphical representations.  the PROV-N, PROV-JSON, and PROV-XML serialization formats, as well as PDF, PNG, and SVG graphical representations.
139  flows of activities (pipelines) which are composed of different activity  flows of activities (pipelines), which are composed of different activity
140  steps, and serialization in VOTable format.  steps, and serialization into the VOTable format.
141
142  This library is currently used in the context of the POLLUX database  This library is currently used in the context of the POLLUX database,
143  which is a stellar spectra database proposing access to high resolution synthetic  which is a stellar spectra database proposing access to high resolution synthetic
144  spectra computed using the best available models of atmosphere and efficient  spectra computed using the best available models of the atmosphere and efficient
145  spectral synthesis codes.  spectral synthesis codes.
146
147  % \articlefigure[width=.6\textwidth]{P9-129_f2.eps}{P9-129_f2.eps}{Pollux Provenance}  % \articlefigure[width=.6\textwidth]{P9-129_f2.eps}{P9-129_f2.eps}{Pollux Provenance}
148
149  When a spectrum is integrated into the database, the provenance information is retrieved and serialized  When a spectrum is integrated into the database, provenance information is retrieved and serialized
150  in different formats and with different levels of detail.  in different formats and with different levels of detail.
151  When a user or a piece of software queries the database via the SSA protocol of the Virtual Observatory,  When a user or a piece of software queries the database via the SSA protocol of the Virtual Observatory,
152  the response contains not only the url of the spectra that correspond to the criteria but also, via to the IVOA DataLink protocol, the URL of a web service  the response contains not only the URL of the spectra that correspond to the criteria, but also, via to the IVOA DataLink protocol, the URL of a web service
153  that delivers the file containing the provenance information in the specified format and for a given level of detail.  that delivers the file containing the respective provenance information in the specified format and for a given level of detail.
154  This functionality has been implemented in the CASSIS spectrum visualization application.  This functionality has been implemented in the CASSIS spectrum visualization application.
155
156  \section{Django package}  \section{Django package}
157  The django-prov\_vo package\footnote{\url{https://github.com/kristinriebe/django-prov_vo}} is an open source Python package that can be reused in Django web applications for  The django-prov\_vo package\footnote{\url{https://github.com/kristinriebe/django-prov_vo}} is an open source Python package that can be reused in Django web applications for
158  serving provenance information. The data model classes are directly mapped to tables  serving provenance information. The data model classes are directly mapped to tables
159  in a relational database. The package provides different interfaces to extract provenance: a REST interface to retrieve lists of entities, activities and agents and a ProvDAL interface, which is defined in the current IVOA Provenance Working Draft. The ProvDAL interface takes the identifier of an entity, activity or an agent as parameter and then returns the available provenance information in one of the serialization formats (currently PROV-N and PROV-JSON are implemented). A few visualization techniques for the retrieved provenance graph are also included.  in a relational database. The package provides different interfaces to extract provenance: a REST interface to retrieve lists of entities, activities and agents, and a ProvDAL interface, which is defined in the current IVOA Provenance Working Draft. The ProvDAL interface takes the identifier of an entity, activity or an agent as a parameter and then returns the available provenance information in one of the serialization formats (currently PROV-N and PROV-JSON). A few visualization techniques for the retrieved provenance graph are also included.
160
161  This \texttt{django-prov\_vo} package was developed for a provenance service of the RAVE\footnote{\url{https://www.rave-survey.org/}} project.  This \texttt{django-prov\_vo} package was developed for a provenance service of the RAVE\footnote{\url{https://www.rave-survey.org/}} project.
162  Within the RAVE (RAdial Velocity Experiment) survey, the spectra of about half a million stars from the southern hemisphere were observed and stellar properties determined.  Within the RAVE (RAdial Velocity Experiment) survey, spectra of about half a million stars from the southern hemisphere were observed and stellar properties determined.
163
164  \section{Prototype PostGreSQL database at CDS}  \section{Prototype PostgreSQL database at CDS}
165  We implemented the IVOA Provenance DM in a test PostGres database at CDS.  We implemented the IVOA Provenance DM in a test Postgres database at CDS.
166  The database handles a small collection of image datasets, such as Schmidt plates, mono-band and color composed images  or HiPS representations of pixel data.  The database handles a small collection of image datasets, such as Schmidt plates, mono-band and color composed images or HiPS representations of pixel data.
167  From the IVOA Provenance Datamodel specification we designed a database schema and implemented the various related tables recommended in the data model as PostGres tables.  From the IVOA Provenance Datamodel specification we designed a database schema and implemented the various related tables recommended in the data model as Postgres tables.
168  % The logical model is shown on Fig. \ref{fig:logicalmodel}.a.  % The logical model is shown on Fig. \ref{fig:logicalmodel}.a.
169
170  % \begin{figure}  % \begin{figure}
# Line 175  Line 175
175  %        \label{fig:logicalmodel}  %        \label{fig:logicalmodel}
176  %\end{figure}  %\end{figure}
177
178  A small set of plates, with their digitization, cutout extractions,  RGB color composition and HiPS generation activities is used to populate the database.  A small set of plates, with their digitization, cutout extractions,  RGB color composition, and HiPS generation activities, is used to populate the database.
179  Various scenarios for querying and displaying the Provenance information have been tested in SQL.  Various scenarios for querying and displaying their provenance information have been tested in SQL.
180  PROV-N, PROV-Json and PROV-VOTable formats are provided for the query response.  For query responses, PROV-N, PROV-JSON, and PROV-VOTable formats are provided.
181  A simple python API allowing users to select the main types of requests and to display the responses via W3C Prov library has been designed.  A simple Python API allowing users to select the main types of requests and to display the responses via W3C Prov library has been designed.
182  It allows users querying for various combinations of Provenance relationships in the database and to visualize the provenance graph in a user friendly representation.  It allows users querying for various combinations of provenance relationships in the database and to visualize the provenance graph in a user friendly representation.
183  % See Fig. \ref{fig:logicalmodel}.b  % See Fig. \ref{fig:logicalmodel}.b
184
185  This provides experience with the DM implementation and clues to build up a TAP SCHEMA representation for Prov-TAP services.  This provides experience with the DM implementation and clues to build up a TAP SCHEMA representation for ProvTAP services.
186
187  \section{UWS Server at Observatoire de Paris}  \section{UWS Server at Observatoire de Paris}
188
189  In the context of the Cherenkov Telescope Array\footnote{\url{https://www.cta-observatory.org/}} (CTA) project, a job control system based on the IVOA UWS pattern has been developed as an open source Python application: OPUS\footnote{\url{https://github.com/mservillat/OPUS}} (Observatoire de Paris UWS System).  In the context of the Cherenkov Telescope Array\footnote{\url{https://www.cta-observatory.org/}} (CTA) project, a job control system based on the IVOA UWS pattern has been developed as an open source Python application: OPUS\footnote{\url{https://github.com/mservillat/OPUS}} (Observatoire de Paris UWS System).
190  This system has been used to test the execution of CTA data analysis tools on a work cluster. It implements the ProvenanceDM concept of ActivityDescription files and provides the provenance information for each executed job in the PROV-JSON and PROV-XML serializations.  This system has been used to test the execution of CTA data analysis tools on a work cluster. It implements the ProvenanceDM concept of ActivityDescription files and provides the provenance information for each executed job in
191    PROV-JSON and PROV-XML serializations.
192
193  The CTA is the next generation ground-based very high energy gamma-ray instrument. Contrary to previous Cherenkov experiments, it will serve as an open observatory providing data to a wide astrophysics community, with the requirement to offer self-described data products to users that may be unaware of the Cherenkov astronomy specificities (see also ADASS Poster 75).  The CTA is the next generation ground-based very high energy gamma-ray instrument. Contrary to previous Cherenkov experiments, it will serve as an open observatory providing data to a wide astrophysics community, with the requirement to offer self-described data products to users that may be unaware of the Cherenkov astronomy specificities (see also ADASS Poster 75).
194
195
196  \acknowledgements This work was partially funded by the Federal Ministry of Education and Research in Germany  \acknowledgements This work was partially funded by the Federal Ministry of Education and Research in Germany
197  and by the ASTERICS project (\url{http://www.asterics2020.eu/}).  and by the ASTERICS project (\url{http://www.asterics2020.eu/}).
198  Additional funding was provided by the INSU (Action Sp\'{e}cifique Observatoire Virtuel, ASOV), the Grand-Sud-Ouest Data Centre,  Additional funding was provided by the INSU (Action Sp\'{e}cifique Observatoire Virtuel, ASOV), the Grand-Sud-Ouest Data Centre,
199  the Paris Astronomical Data Centre and the Observatoire Astronomique de Strasbourg.  the Paris Astronomical Data Centre, and the Observatoire Astronomique de Strasbourg.
200
201
202  \bibliography{P9-129}  % For BibTex  \bibliography{P9-129}  % For BibTex

Legend:
 Removed from v.4595 changed lines Added in v.4596