\documentclass[10pt,a4paper]{ivoa}
\input tthdefs

\usepackage{color}
\usepackage{todonotes}
\usepackage{listings}
\lstloadlanguages{sh,make,[latex]tex,XML,XSLT}
\lstset{flexiblecolumns=true,numberstyle=\small,numbers=left,
  identifierstyle=\texttt}


\usepackage{color}
\definecolor{gray}{rgb}{0.4,0.4,0.4}
\definecolor{darkblue}{rgb}{0.0,0.0,0.6}
\definecolor{cyan}{rgb}{0.0,0.6,0.6}

\lstset{
  basicstyle=\ttfamily,
  columns=fullflexible,
  showstringspaces=false,
  commentstyle=\color{gray}\upshape
}

\lstdefinelanguage{XML}
{
  morestring=[b]",
  morestring=[s]{>}{<},
  morecomment=[s]{<?}{?>},
  stringstyle=\color{black},
  identifierstyle=\color{darkblue},
  keywordstyle=\color{cyan},
 % morekeywords={xmlns,version,type}% list your attributes here
}

\hyphenation{name-space}

\usepackage{todonotes}

\usepackage[utf8]{inputenc}

\definecolor{texcolor}{rgb}{0.4,0.1,0.1}
\newcommand{\texword}[1]{\texttt{\color{texcolor} #1}}

\iftth
  \newcommand{\BibTeX}{BibTeX}
\fi

\iftth
 \newcommand{\comicstuff}[1]{
    \begin{html}<span class="comic">#1</span>\end{html}}
\else
  \newcommand{\comicstuff}[1]{(HTML exclusive material)}
\fi
\customcss{custom.css}

\title{XML Schema Versioning Policies}

\ivoagroup{Standards and Processes}

\author[http://www.ivoa.net/cgi-bin/twiki/bin/view/IVOA/PaulHarrison]{Paul Harrison}
\author[http://www.ivoa.net/cgi-bin/twiki/bin/view/IVOA/MarkusDemleitner]{Markus Demleitner}
\author[http://www.ivoa.net/cgi-bin/twiki/bin/view/IVOA/BrianMajor]{Brian Major}
\author[http://www.ivoa.net/cgi-bin/twiki/bin/view/IVOA/PatDowler]{Pat Dowler}

\editor{Paul Harrison}

\previousversion{v1.0}
       
\begin{document}

\SVN$Rev$
\SVN$Date$
\SVN$URL$

\begin{abstract}
This note describes the recommended practice for the evolution of IVOA
standard XML schemata that are associated with IVOA standards.  The criteria for
deciding what might be considered major and minor changes and the policies for dealing with each case are described.
\end{abstract}



\section*{Acknowledgments}

The content of this note is derived from discussions that occurred in a splinter
session at the June 2015 IVOA interoperability meeting in Sesto, Italy.

\section{Introduction}

Many of the standard protocols and data models developed by
the International Virtual Observatory Association (IVOA) have used XML
\citep{std:XML} for message or object serialization. The structure of
these XML files has
usually been constrained using the XML schema definition language
\citep{std:XSD}, or XSD for short.
The particular schema that has been associated with a standard 
is defined by the ``target namespace'' (hereafter refered to as simply ``the
namespace'') of the schema -- the namespace identifier itself is a URI that
typically has a form that contains the version number of the standard. There
exist in many programming languages XML parsers that can use the XSD schema
automatically to do a strong check whether an instance of an XML
document conforms to the given schema. If the XML document does conform to the
structure defined by the schema then it is known as ``valid'' and even a small
deviation from the specified structure will mean that the XML instance is
``invalid''. This strong check of validity is extremely useful in the context of
interoperating services and clients within the VO as it guarantees that both
sides of an interaction will agree on the structure of the document and
hence its interpretation. To maintain
this behaviour, the conventional XSD practice is to give the schema a new
identity (namespace) if any changes are made to it so that both client and
server can agree on the exact version of a schema that they are using for
checking validity of an XML instance.

Once a VO service has been standardised there will typically be a growing number
of clients that are coded against the particular version of the
schema -- any changes to the definition of the schema that is subsequently used
by a new version of the service to create instance documents will result in
immediate classification as ``invalid'' by the clients that have not themselves being
updated to use the new schema.
As explained above, in general this property is desirable for guaranteeing interoperability, but it does limit
the ability even to correct errors in the original schema definition without
causing disruption to the deployed clients. 

A related problem occurs because, to namespace-aware XML parsers, the
element name is a tuple of the namespace URI and the tag content.  For
instance, in the XML document

\begin{lstlisting}[language=XML]
<doc xmlns="http://example.com/1.0"/>
\end{lstlisting}
the fully qualified name of the element includes the namespace and is
conventionally written as \verb|{http://example.com/1.0}doc|, and clients will expect this full
form, whether or not they perform schema validation.  As soon as the
namespace changes, any client expecting this element name will no longer
understand anything in the document.  Rather typically, however, the
client would still be able to make sense of the document as far as it is
relevant to the client if it ignored the namespace part of the element
name.  In consequence, many clients started discarding the namespace
part entirely, which then leads to new interoperability problems, for
instance, when elements from a different namespace are embedded within
such a document.  Hence it is desirable to limit namespace changes to
cases when legacy clients would definitely break.

This note describes the
circumstances in which it is permissible to make changes to a schema and not
change its namespace.


\section{Schema Versioning}
It is the case that two XML instances are formally regarded by XML
conventions as not equivalent if the only textual difference between them is
that the namespace declaration is not the same. We
can use the observation that simply removing the namespace definition from both instances
would make the instances both textually the same and equivalent in the strict
XML sense to inform the class of changes can be to the schema definition without
necessarily needing to change the namespace, and how a client should treat such
changes.
\subsection{Minor changes}
In general the class of changes that might be considered minor are those
which allow legacy clients (i.e. without rewriting) to keep functioning with
content produced against the new schema.
Specific structural changes to the schema that allow this goal to be achieved
include:

\begin{itemize}
  \item Not removing concepts (i.e. elements or attributes) from the old
  schema.
  \item Ensuring that any new concepts are optional.
\end{itemize}
Even with the restrictive conditions above it is still necessary that any
consumer of XML instance documents takes the approach that it does not do strict
schema validation against the version of the schema that it knows about, but
rather ignores everything that it does not understand. This approach is allowable
because any new concepts are optional even for consumers of the XML instance
that are aware of the latest version of the schema, and so clients cannot use
this information for fundamental changes in the behaviour of an IVOA protocol.
In other words the IVOA protocol remains backwards compatible and would only
warrant a ``point change'' in the standard version.

Conversely if the conditions above cannot be met by an evolution of an IVOA
standard then it is a indication that the standard is undergoing a ``major
change'' and backwards compatibility may be broken. Such a change would
involve both a change in the first part of the standard document's version
 number as well as a change in the version used in the schema
namespace.
\subsubsection{Determining if the changes are indeed minor}
Although the conditions outlined above for minor changes should generally be
strictly adhered to in the design of minor extensions to standards, there are
occasions where a new version of a standard might try to correct an error made
in a previous version. For example a certain construct could have been
schema valid which should not have been allowed in a correctly authored
schema that expressed the intentions of the standard. In this case it is likely
that the correction would break the first of the above conditions, but hopefully
clients would not be written to expect the unintentionally valid construct and
would not be unduely impacted by the change. It would be up to the IVOA
Technical Coordination Group (TGC) to determine the likely impact of such a
change and whether the violation of the constraints for a minor change would be
allowable in such circumstances.

\subsection{Indicating the version number}

When allowing minor changes to the schema without changing the
namespace it becomes imperative that the the minor version number of the schema
is communicated via a means other than the namespace.  This is true for
both instance documents and schema documents. However, the exact usage in each
case means that the semantics of the ``version'' are subtly different;
\begin{itemize}
  \item In instance documents it is the overall version of the service/protocol
  that is being conveyed.
  \item In the \xmlel{schema} element itself it is the connection with the
  accompanying standard document that should be represented.
\end{itemize}

This means that the textual content of each version will be different, as
illustrated below.

\subsubsection{Version numbers in instance documents}

It is important that
clients are able to know which version of the schema was used to create an XML
instance.  A common use case envisioned is that
this information can be used to discover a service's ability to
correctly respond to newly specified parts of the protocols. 

Hence, in conforming documents (e.g., service responses), 
root elements must have an
attribute giving the value of the corresponding \textit{major.minor} number of the
accompanying standard document.

The recommended local name of this attribute is
\xmlel{stdVersion}. Previously the recommended name had been \xmlel{version},
but this new name has been chosen to emphasise the difference in semantic role
that the version takes in instance documents. Existing schema that use
\xmlel{version} should not be considered in need of update because of this new
recommendation.

When global elements from a different schema are included within other
XML files, each of these global elements should have a version attribute
of its own.  Where the discovery of extended capabilities is not a
likely use case, such version attributes may be left out.

As an example, consider a VOSpace 2.1 transfer element within a UWS 1.1
job element:

\begin{lstlisting}[language=xml]
<uws:job 
    xmlns:uws="http://www.ivoa.net/xml/UWS/v1.0" 
    version="1.1">
  <uws:jobId>sv_ex</uws:jobId>
  ...
  <uws:jobInfo>
    <vos:transfer 
        xmlns:vos="http://www.ivoa.net/xml/VOSpace/v2.0"
        version="2.1">
      <vos:target>vos://example.com!vospace/mydata1</vos:target>
      ...
  </uws:jobInfo>
</uws:job>
\end{lstlisting}

Note how both global elements in the example (\xmlel{uws:job} and
\xmlel{vos:transfer}) have a \xmlel{version} attribute.  This example
also illustrates how with legacy namespace URIs, the minor version
apparently indicated by the namespace URI is different from the actual
minor version relevant as declared using the \xmlel{version} attribute.

\todo{should anything be mandated about the style of stdVersion attribute or is
it ok to do how you want}

\subsubsection{Version numbers in schema files}
It is of similar importance to software writers to know exactly the version of
the schema that they are using. In the XSD definition there is an appropriate
\xmlel{version} attribute on the top level \xmlel{<schema>} element that can be
used for this purpose.

\begin{lstlisting}[language=XML]
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
   targetNamespace="http://www.ivoa.net/xml/UWS/v1.0"
   xmlns:uws="http://www.ivoa.net/xml/UWS/v1.0"
   xmlns:xlink="http://www.w3.org/1999/xlink" 
   elementFormDefault="qualified"
   attributeFormDefault="unqualified"
   version="1.1-PR-20150626"
>
\end{lstlisting}

This version attribute is not used formally by the schema validation machinery
but can be used as desired here to indicate a precise version of the schema
given that the namespace is only used to differentiate between major revisions
of the schema. In general a new schema should only be published in tandem with a 
new version of a standards document or an erratum to such a document, so this 
particular \xmlel{version} attribute can be seen as a strong tie to the precise
standard document rather than the less fine-grained overall standards versioning
scheme.
\todo{should say something about the formatting of this if as a result of
erratum}

\subsubsection{Version numbers in Namespace URIs}

In the IVOA, namespace URIs traditionally contained the full version
specification, as in \nolinkurl{http://www.ivoa.net/xml/UWS/v1.0}.
This practice is somewhat paradoxical, as changes in the minor version
number should, by the IVOA's document standards
\citep{2010ivoa.spec.0413H}, not break downwards compatibility, but a
change in the namespace URI is, as argued above, incompatible in both
directions.

Therefore, future namespace URI must only contain the major version
number.  The recommended format is to have the major version as the last
URI element, for instance,

$$\hbox{\nolinkurl{http://www.ivoa.net/xml/UWS/2} .}$$

An unfortunate side effect of the proposal made in this note is
that legacy namespace URIs become somewhat confusing, as a namespace
like \nolinkurl{http://www.ivoa.net/xml/UWS/v1.0} can be defined by
schema documents with actual versions 1.1, 1.2, etc.  

Specifications employing such legacy namespace URIs should warn their
readers to this effect.  The suggested presentation is an admonition box
with a content like this:

\begin{quotation}
The namespace URI \nolinkurl{http://www.ivoa.net/xml/UWS/v1.0} is used
for backwards compatibility.  The presence of the string \texttt{v1.0}
does \emph{not} indicate that the document content conforms to version
1.0 of the schema, or can be validated against that schema.  It does
indicate, however, that documents valid according to version 1.0 of the
schema still validate with the current schema file.  It also means that
any XML reader compliant with the recommentations of in Harrison et al
(2016) [this would be a reference to the current version of this note]
must be able to process the elements' content if they support any
version 1.0 or later of this specification.

The version an instance document actually conforms to can be obtained
from the \xmlel{stdVersion} attribute on the instance document's root
element [or the element containing the namespace declaration, if
applicable].  
\end{quotation}



\section{Hosting the Schema on the IVOA Web Site}
As mentioned in the introduction the schemata are typically associated with a
particular version of a standard and so are typically included in an appendix of
the standard. In addition they are located on the ivoa website
\url{http://www.ivoa.net/xml} to allow software authors interactively to
obtain the latest version directly as a file.

The availability of the schemata via the IVOA web site also
allows XML parsers to be instructed to fetch the schema automatically when
validating an XML instance. This is done using the \xmlel{xsi:schemaLocation}
attribute which pairs a namespace with the URL of a schema that has definitions
in that namespace. Although there is no particular necessity to do so, it has
been standard practice to use namespaces that have a one-to-one correspondence
with a location on the IVOA web site. The advantage of this approach is that the
namespace is ``owned'' by the IVOA and there is therefore much less chance of
accidental clashes of namespaces.

Thus a current schema is usually available at two URLs on the IVOA site;
\begin{enumerate}
  \item at a URL that corresponds to the namespace \newline e.g.
  \texttt{http://www.ivoa.net/xml/UWS/v1.0}
  \item at a URL that corresponds to the filename of the schema \newline e.g.
  \texttt{http://www.ivoa.net/xml/UWS/UWS-v1.0.xsd}
\end{enumerate}

As this note now recommends that there are potentially many minor versions of a
schema all in the same namespace, the location that the namespace URI
redirects to must be updated when a new minor version is published. The
various minor versions of a particular schema should be available at the
``filename'' style URLs of the second kind. It should be noted that the
actual file pointed to by the ``namespace'' style URL should only be updated to
point to the new minor version when the associated standard has reached the
``recommendation'' stage.

\section{Client use of schema}
Using this strategy means that client software that simply uses validation with
the parser utilizing this schema location hint will always pick up the latest
version of the schema, which should mean that the instance document will always
be valid assuming that the server has created a strictly valid document. Note
that this still applies even if the server is using an older (minor) version of
the schema, as new versions are only allowed to add optional features to the
schema.

It is often the case that client software is written to use a local copy of the
schema to avoid the overhead of continually downloading the schema. In this case
it is possible that the XML instance document will not be valid against the
local schema if the server is working against a newer minor version of the
schema -- as it could have added a new optional element into the XML instance.
It is for this reason that the client side of an IVOA service should be
forgiving in ignoring elements that it is not expecting. It is possible for the
client to recognise that the XML instance is valid against a newer version of
the schema by comparing the version attribute of the root element of the XML
instance against the version attribute in the local XSD schema document.

\section{Summary of Recommendations}
This section summarizes the main recommendations contained within this document.

\begin{itemize}
\item The namespace URI should contain only the associated standard major number, 
and therefore does not change when a minor version increment occurs.
e.g.\ \texttt{http://www.ivoa.net/xml/ABC/v1}. However, existing namespace URIs
which contain a ``\texttt{.0}'' minor increment suffix should not be changed to
conform to this rule.

\item Include a \xmlel{stdVersion} attribute in the top level element to allow
client version discovery -- this version number should include the full standard version
number including the minor version increment. -- 1.1

\item Set the \xmlel{stdVersion} attribute of the \xmlel{<schema>} element in the
XSD to be equal to the full standard version including the minor version
increment and document status and version -- e.g.\ 1.1-WD-20150607

\item Minor version changes should not break clients -- i.e. in general they
should only add new elements/attributes, not remove formerly valid
content --
correspondingly, this means that clients should quietly ignore things that they
do not know about in the XML -- i.e. they should not automatically issue an error
if schema validation fails (though of course if they can determine that the
validation error is because of a missing required element then they should
issue an error).
\item In contrast servers must produce strictly valid documents, and service
validators must test strict validity against the relevant schema (discovered
from the namespace and the version element).
\end{itemize}

\appendix


\section{Changes from Previous Versions}

\subsection{Changes from version 1.0}
% these would be subsections "Changes from v. WD-..."
% Use itemize environments.
\begin{itemize}
  \item recommend that instance document version attribute be named 
  \xmlel{stdVersion} in the namepace of the parent schema rather than
  \xmlel{version}.
  \item emphasise how in the different usages, ``version'' can have different
  semantic meanings and textual representation.
\end{itemize}

\bibliography{ivoatex/ivoabib,ivoatex/docrepo}


\end{document}
