ViewVC logotype

Contents of /trunk/projects/WebAssets/WebAssets.tex

Parent Directory Parent Directory | Revision Log Revision Log

Revision 2950 - (show annotations)
Thu May 7 15:27:48 2015 UTC (6 years, 5 months ago) by marco.merot@gmail.com
File MIME type: application/x-tex
File size: 24389 byte(s)
webassets: changed assets drawing, added mailing list maintenance and
repo structure issues (Marco)

1 \documentclass{ivoa}
2 \input tthdefs
4 \usepackage{soul}
6 \title{IVOA Web Assets\\architecture and maintenance}
8 \ivoagroup{\textit{not applicable}}
10 \author{Bruce Berriman}
11 \author{Marco Molinaro}
12 \author{Sarah Emery Bunn}
13 \author{Giulia Iafrate}
14 \author{Mark Bartelt}
15 \author{Santosh Jagade}
16 \author{Francesco Cepparo}
17 \author{Giuliano Taffoni}
18 \author{Claudio Vuerli}
19 \author{Sharmad Navelkar}
21 \editor{Bruce Berriman, Marco Molinaro}
23 % \previousversion[????URL????]{????Funny Label????}
24 \previousversion{This is the first public release}
26 \begin{document}
27 \begin{abstract}
28 The IVOA makes use of a documentation repository, a web site, a collaborative Wiki, and a mailing list. These assets are central to fostering collaboration between the IVOA partners and for creating records of collaborative efforts within the alliance. These services are hosted and maintained by one or more of the IVOA partners on behalf of the Alliance as a whole. This Note describes these services, and the critical features of their architecture, operation and maintenance. It also describes the connections and interoperability between them. The Note is intended as a guide to organizations taking on the role of hosting one or more of these services.
29 \end{abstract}
31 %\section*{Acknowledgments}
32 % Placeholder currently: if we'll need it we'll keep, otherwise delete it.
34 \section{Introduction}
35 The IVOA has a website where information about standards and protocols, events, documentation are located for dissemination to IVOA members, astronomers and the general public. This website is associated with the domain name \textit{ivoa.net} and its web content was revised as described in \citet{lawrence10} a few years ago.
37 The web page is one of a number of IVOA assets used for maintaining records and fostering collaboration: the others are a document repository, collaborative wiki pages and a mailing list service.
39 This Note describes the essential technical aspects of the web pages (Sec.~\ref{sec:web}), document repository (Sec.~\ref{sec:doc}), wiki (Sec.~\ref{sec:wiki}) and mailing list (Sec.~\ref{sec:mailman}) that an IVOA member planning to install, operate and maintain any of these services should know about. The Note is based on the experience gained by its authors during the hand over of the IVOA web assets from Caltech (as a member of US VAO national project) to INAF-OATs (VObs.it) and IUCAA (VO-India). The Note is intended also as a guide in planning future transitions of the assets, and the Appendix describes critical issues that may be encountered, and possible solutions to them.
41 Future changes and modifications to the services that are part of the IVOA web assets should be reflected in (minor) revisions of this Note.
43 \section{Domain Name \& Subdomains}\label{sec:domain}
44 The IVOA web content is built up as a set of services (described in the Sections \ref{sec:web}, \ref{sec:doc}, \ref{sec:wiki}, \ref{sec:mailman}) that are subdomains of the main \url{ivoa.net} domain. The domain name is currently registered and managed by VObs.it (actual management is based in INAF-OATs\footnote{\url{www.oats.inaf.it}, contact ivoadoc@ivoa.net for further details}) after the hand-over from VAO (management located at Caltech); the domain registrar is \url{http://godaddy.com/}. A second domain, \url{ivoa.info}, exists and is currently registered but it is not used.
46 A block diagram of the current architecture of the web assets and their associated domain names is depicted in Fig.~\ref{fig:webassets}. A critical feature of the architecture is the interconnectivity between the web pages, the document repository, the wiki content and the mailman archives and its subscription system. When the services are hosted on the same server, linking is straightforward and linear, but when the services are mounted on distributed servers, as they are now and is likely the case in the future, linking is more complex and hosting organizations must be aware of caveats in redirections and aliasing of server names. The check lists in Appendix~\ref{app:chklist} is intended to help keep all of these connections tight when they are moved.
48 \begin{figure}
49 \centering
50 \includegraphics[width=\textwidth]{ColdBackup.png}
51 \caption{Quicklook architecture of IVOA Web Assets. Interconnections between web pages, wiki pages, document repository and mail archives are not included. Partners managing the various services are color-coded as in the legend. The backup system is not yet in place at the time this Note is issued.}
52 \label{fig:webassets}
53 \end{figure}
55 Here is the list of the subdomains for \url{ivoa.net} as of this writing:
56 \begin{itemize}
57 \item \textit{www}, hosting the web pages described in Sec.~\ref{sec:web};
58 \item \textit{wiki}, pointing to the IVOA web of the TWiki installation (Sec.~\ref{sec:wiki});
59 \item \textit{mail}, is the mail server for all ivoa.net addresses (see Sec.~\ref{sec:mailman} for details) and keeps the Mailman archives for the IVOA mailing lists;
60 \item \textit{rofr}, pointing to the IVOA Registry of Registries \citep{note:rofr};
61 \item \textit{skos}, pointing to the SKOS vocabularies (no longer active?).
62 \end{itemize}
64 This list is taken from the DNS zone file from the DNS server at INAF-OATs, the primary name resolver for the \url{ivoa.net} domain. Secondary (backup) DNS-es for the domain are hosted by the Trieste University Information System Division and the INAF Data Elaboration Center (located in Bologna, Italy). This should secure hostname redirection.
66 The two services at the end of the list (\textit{rofr} and \textit{skos}) are described in the following subsection (Sec.~\ref{subsec:rofrskos}) due to their specific roles in the context of the IVOA.
68 \subsection{Registry of Registries and SKOS services}\label{subsec:rofrskos}
69 Both \url{rofr.ivoa.net} and \url{skos.ivoa.net} are resolved through the DNS zone. This means that the responsible partner for the domain registration is also responsible for the correctly pointers to these two services. However, these two services are not located in the VObs.it--VO-India connected environment. The RofR is managed by NCSA (as part of the US VAO initiative) while the skos service seems to be currently dead.
71 \subsection{Hand-over Strategy}\label{subsec:strat}
72 Since moving the IVOA web content requires both relocating the DNS server and zone file and transferring the content of the services themselves, the hand-over from one organization to another should be carefully planned and scheduled to minimize downtime.
74 Based on our experience in moving services from Caltech, the recommended plan is as follows: first, move the DNS zone and then, once name resolving is performed by the new manager, move the various services one by one, modifying accordingly the DNS zone to point to the new IP addresses.
75 This plan means that only one service is down at a time, and bugs and errors resulting from the services' transfer must be largely due to the relocation itself.
77 Appendix~\ref{app:chklist} provides a comprehensive checklist for future transfer of individual services and of the complete set of assets.
79 \section{Web Pages}\label{sec:web}
80 The IVOA web pages are now hosted at IUCAA, by VO-India, while the Wiki and Mailman related links will be redirected to the TWiki installation that has been moved to INAF-OATs (managed by VObs.it). Current setup of IVOA web pages is based on HTML \& CSS running on an Apache 2 web server and is dedicated to hosting the IVOA web pages \& the document repository. During the transfer and set up of the IVOA web pages at IUCAA, many internal links were redirecting to the previous IVOA server located at VAO \, and these have now been mapped to the respective internal references. In addition the image (QUESTION: what image is this? Are there words missing? ) \& CSS references have also been modified.
82 The IVOA web pages will be updated to support events such as the following:
83 \begin{itemize}
84 \item An upcoming Interoperability Meeting;
85 \item Release of a new issue of the IVOA newsletter;
86 \item New countries joining the IVOA.
87 \end{itemize}
88 All these structural changes have to be validated or approved by the IVOA Executive Committee or the IVOA Executive Chair. The IVOA website templates can found at \url{http://www.ivoa.net/Templates/}.
90 \subsection{Newsletter}\label{subsec:news}
91 Within the IVOA web pages, there is a page dedicated to hosting the biannual newsletter edited by the IVOA (\url{http://www.ivoa.net/newsletter}). The editing and management of this newsletter is performed by SAO, and the released issues are hosted on the web pages managed by IUCAA.
93 \section{Document Repository}\label{sec:doc}
95 Along with IVOA web pages, IUCAA hosts the IVOA document repository. This repository can be accessed through \url{http://www.ivoa.net/documents/index.html}. This page is further sub-divided into:
96 \begin{itemize}
97 \item Technical Specifications
98 \item Notes
99 \item Document promotion process summary
100 \item Submission Log
101 \end{itemize}
103 The PR (Proposed Recommendation) and REC (Recommendation) documents have to be accepted through the procedure described in \cite{std:docSTD}.
104 The REC (Recommendation) process can usually be followed on the exec mailing list.
105 In contrast, The WD (Working Draft) and Notes can be published and updated without further approval.
107 IVOA members can upload a new or updated document using the form at \url{http://www.ivoa.net/cgi-bin/up.cgi} and any new document request will arrive on the IVOA Document Coordinator (hereafter DC) email (\url{ivoadoc@ivoa.net}). Once the DC completes the process of acceptance, the submitter will receive an mail from her for verifying the published documents and content.
109 \subsection{Document Coordinator}\label{subsec:doccoord}
110 Before the switch from VAO to VObs.it and VO-India, the IVOA Document Coordinator was co-located geographically with the IVOA web server. This obviously made the DC's role easier to perform. Now though, the current DC is based in Trieste. While this has required that she be provided with an account for remote access to the web and repository server in IUCAA, she requires only a local connection to the TWiki and Mailman servers.
112 \section{Collaborative Wiki}\label{sec:wiki}
113 The collaborative wiki site for the IVOA community is powered by a TWiki\footnote{\url{http://twiki.org/}}, and currently the IVOA uses the TWiki 5.1.1 engine. The hand-over of this service mainly required a packaged backup of the web deployed structure of the TWiki installation and some reconfiguration at the Trieste site.
115 The IVOA wiki site has some peculiarities. There is an \textit{IVOA} web that is used in place of the default \textit{Main} web of TWiki. The \textit{pub} folder, used for all of the TWiki topics attachments, is mapped to \textit{internal} both on the wiki and the web sites.
117 As a performance matter, the TWiki installation at the INAF-OATs site (managed by IA2 data center staff) hosts only the IVOA TWiki webs and is located on a server that is fully devoted only to \url{ivoa.net} services. This is not needed in principle, but is clean considering the Mailman (see Sec.~\ref{sec:mailman}) and underlying mail server setup.
119 \section{Mailing Lists}\label{sec:mailman}
120 Together with the wiki, the mailing lists of \url{ivoa.net} are used for collaboration, information and coordination. The mailing lists are managed through the Mailman\footnote{\url{http://www.gnu.org/software/mailman/}} application. As with the wiki, the hand-over of the mailing list service required a packaged backup of the main folders of the application installation, as well as checks for mailing lists changes between transfer and installation at the new site.
122 The Mailman application requires a mail server to dispatch mail through and get incoming mail from. Any existing mail server will fit if it is configured to manage the \url{ivoa.net} mail domain. However at INAF-OATs a dedicated mail server has been setup to handle the \url{ivoa.net} mail domain (and consequently the mailing lists), to prevent overload of the local mail servers. This server runs on the same machine as the wiki.
124 \section{Web Assets Cold Backup}\label{sec:backup}
125 A \textit{cold backup} (that will be provided by the China-VO partner) is under definition to guarantee service continuity during maintenance or switch-over of the web assets. A sketch of the backup system is draft in Fig.~\ref{fig:webassets}.
127 The proposal (as of this writing) from VObs.it is the following:
128 \begin{itemize}
129 \item for each of the current two servers in production (one in Italy, the other in India)
130 \begin{itemize}
131 \item prepare a sibling server in China,
132 \item keep it up-to-date (software)
133 \item keep it synchronized (content)
134 \end{itemize}
135 \end{itemize}
136 In this way, should it be needed to move or change one of the production services, the DNS zone can be changed accordingly and in proper time (the needed time to propagate the change through DNS-es) have the cold backup server keep up.
138 When the move/update is in place, the reverse procedure will take place.
140 Upgrading and synchronization of the master (production) and slave (cold backup) servers should be handled by the service maintainers, i.e. VObs.it for wiki and mail, VO-India for web and document repository, with China-VO responsible for keeping the servers online and ready.
142 %\section{Conclusions/Summary?}
143 %\hl{M.Mo: do we need some summary and/or conclusions?}
145 \appendix
146 \section{Assets transfer check list}\label{app:chklist}
148 The listings below report the main (and/or minor) items to consider when planning to move one or all the IVOA web assets services to new locations.
149 The suggested approach in the case of a full transfer, as explained in Sec.~\ref{subsec:strat}, is to move first the DNS server to the new location(s) and subsequently move the services one by one, changing the DNS zone file after verifying the operational statis of the new service instances.
151 \textbf{NOTE 1}: Currently the web pages link to Twiki with a double mapping on \url{twiki/bin} and \url{cgi-bin/twiki/bin/} for historical reasons. Current mailing list archives are reachable at \url{mail.ivoa.net/mailman}, but calls from \url{www.ivoa.net/mailman} are also redirected there, a consequence of earlier assets transfer.
153 \textbf{NOTE 2}: It is recommended that software installations be as standard as possible to simplify upgrades, updates and backups. Currently both INAF-OATs and IUCAA provided machines run the Linux CentOS (version 6.5 final) server.
155 \subsection{Domain Name and Zone}\label{subsec:chkdns}
156 \begin{itemize}
157 \item Contact the responsible person for the domain to define the schedule for the DNS zone hand-over (e.g. through the IVOA Document Coordinator)
158 \begin{itemize}
159 \item Currently contact C.~Vuerli at INAF/OATs
160 \end{itemize}
161 \item Coordinate with the previous domain owner in moving the ownership of the domain and thenpoint to the new DNS server
162 \begin{itemize}
163 \item Define domain registrar details (currently accounts on \url{GoDaddy.com})
164 \end{itemize}
165 \item Prepare the new DNS server with a twin zone file and ensure there is a backup, preferably in geographically different location; requirements
166 \begin{itemize}
167 \item 1 physical file transfer
168 \item 2 or more DNS servers available (currently 1 master and 2 spare)
169 \end{itemize}
170 \item Switch the DNS with the registrar
171 \end{itemize}
173 \subsection{Web Pages \& Document Repository}\label{subsec:chkweb}
174 \begin{itemize}
175 \item Apache
176 \begin{itemize}
177 \item Get current httpd.conf files to keep resource mapping
178 \item Perform httpd.conf redirects
179 \end{itemize}
180 \item Python (This may need more info? Does this mean load python? if so which version?)
181 \item Request the current IVOA Document Coordinator for her/his duties manual (includes some management passwords)
182 \end{itemize}
184 \subsection{Wiki}\label{subsec:chkwiki}
185 \begin{itemize}
186 \item TWiki-5.1.1 (build 22570 14.Jan.2012), Plugin API version 1.4
187 \begin{itemize}
188 \item Patched for Security Alert CVE-2014-7236
189 \item Check Perl environment and \textit{cgi-bin} scripts shebang
190 \item Disable automatic user registration form
191 \item Get the (currently $\sim$7GB) TWiki dump and deploy it
192 \end{itemize}
193 \item Apache
194 \begin{itemize}
195 \item Version 2.x recommended
196 \item \textbf{Warning}: don't use Apache tool for .htaccess files!
197 \item Configure TWiki httpd.conf
198 \item Redirect \url{http://wiki.ivoa.net} root context to \url{http://wiki.ivoa.net/twiki/bin/view/IVOA}
199 \item Remember that TWiki \textit{pub} directory is mapped to \textit{internal} in IVOA wiki site
200 \end{itemize}
201 \end{itemize}
203 \subsection{Mailing Lists}\label{subsec:chkmailman}
204 \begin{itemize}
205 \item Have a mail exchanger ready to manage the \url{ivoa.net} mail domain (currently: postfix)
206 \item Setup a default mailman installation connected to that mail server
207 \item Get the (currently $\sim$2.5GB) mailman archive, data and lists dump
208 \item Merge it to the new mailman server
209 \end{itemize}
211 \section{Issues based on the 2014 hand over}\label{app:issues}
213 This section lists the issues that arose in the process of transferring the IVOA web assets from Caltech to IA2 and IUCAA. It is anticipated that solutions to these issues or changes in methodology resulting from them will be reflected in updates to this Note.
215 A general feel about the full web assets of the IVOA is that they need to be polished for easier and cleaner hand-over. As they are now, the transfer is not completely straightforward and configurations of the TWiki and its connection back and forth with the web, mail archives and repository services are quite complex. Even simplifying the services to be able to move them in a simple packaging and unpackaging scenario will require ad hoc configuration to retain backwards-compatibility, but would make it easier to move the services as if they are a single package. This issue can affect also the idea of mirroring the ivoa.net services for backup or to ensure continue availability of the services.
217 \subsection{Mailing list issues}
218 \subsubsection{Posts from non-members to a list}
219 A big issue with the mail streams concerns Spam. With a fresh configuration, the mail server for \url{ivoa.net} limits the amount of direct spam, but doesn't prevent Spam posted by non-members.
220 This flood doesn't reach the list members because only members of one list are allowed to post to that list. However this behavior of supervised \textit{deferring} post transmission has the drawback that the IVOA-Doc needs to filter through tens or hundreds of mail daily to search for false positive non-member posts flagged as spam.
222 It would probably be easier to discard directly those posts. This has the drawback that a non-member (or a member that uses a secondary email address) that sends a post will never know her/his post has been lost, but has the advantage to highlight to IVOA-Doc real bounces or moderated posts issues.
224 \subsubsection{interop@ivoa.net list}
225 Between the IVOA mailing lists there's the \url{interop@ivoa.net}. This list has the advantage to be a general way to inform all the VO community across all working groups or tasks.
227 It seems however that it is not a super-container of the other lists, thus failing to reach the full community. This may be caused also by the fact that one has to reach the bottom of the mailing lists subscription page (\url{http://ivoa.net/members/}) to find this list.
229 A two step approach (or two solutions) can solve this issue:
230 \begin{itemize}
231 \item Moving the interop@ivoa.net subscription at the top of the page, specifying its meaning;
232 \item Automatically subscribing to the interop lists, whoever subscribes to one of the other lists (except the \textit{role} lists).
233 \end{itemize}
235 \subsubsection{ivoadoc@ivoa.net}
236 Moving the IVOA services from a single location to multiple ones has led also to a minor issue related to the \url{ivoadoc@ivoa.net} mailing list. This list was a mail alias previously but now, given there is need to have technical support for the IVOA-Doc from two sites, this list has really become a mail dispatcher (4 people are currently listed there).
238 This is actually a problem because this alias forwards mails that are of no interest for the technical support to the Document Coordinator. We suggest that it may be better to use \url{ivoadoc@ivoa.net} for the IVOA-Doc only and setup an \url{ivoadoc-tech@ivoa.net} address, or similar, for the technical support.
240 \subsubsection{Lists or aliases}
241 The \url{ivoadoc@ivoa.net} example above shows the usage of a mailing list instead of a mail alias (or a small distribution list). If the IVOA Mailman application is connected to a mail server that serves multiple domains, this can simplify the maintenance of lists and aliases by putting everything into lists (even if some of the lists contain only one person).
243 Currently, after the assets transfer, Mailman for the IVOA operates on a dedicated mail server, so probably it could be cleaner to use lists where needed and aliases where lists (with archiving, moderation and so on) are not needed. It may also solve some \textit{redirect} issues such like \url{ws@ivoa.net} having as the only member \url{grid@ivoa.net}.
245 \subsubsection{Members maintenance and Aliasing}
246 One of the most demanding activities required for the IVOA Document Coordinator is the check for:
247 \begin{itemize}
248 \item Mail bounces due to non-members posting to the lists (that are members-only configured);
249 \item Spam drilling the mail moderation posts;
250 \item Members that, for mail dispatching reasons, fall into the moderated or bounce, zone of Mailman.
251 \end{itemize}
252 Avoiding these activities is impossible, as the only alternative is to completely close the lists to non-registered emails, thus avoiding spam-related bounces and highlighting only the registered-members posts. This solution, however, has the drawback of losing any mail sent from members that inadvertently send mails from an email address different from the registered one.
254 Multiple emails for the same user is also a (minor) issue, leading to multiple registrations of the same user.
256 \subsubsection{Mailing Lists Moderation}
257 It is not clear whether the role of moderator of the various IVOA mailing lists is an IVOA Doc only one or if the role of bounces and moderation should be shared with chair persons the lists refer to.
259 Obviously adding the moderator role to WG/IG chairs/vice-chairs will increase the complexity of maintenance of the mailing system.
261 \subsection{Documents Repository Structure}
262 There seems to be no clear guidelines on the way the various documents are spread within the IVOA document repository. Recommendation, Proposed Recommendations, Working Drafts, Notes, are all submitted through the same mechanism, leaving the IVOA Doc a few elements to understand which location the newly submitted document landing page should be located.
264 Usually it's easy identified, e.g. by the Working Group identifier or by the fact that it's a Note, but this doesn't prevent to have a structure that does not allow for a simple inference of the location out of the standard name.
266 There exist documents that live in the documents root base, others subdivided in WD, PR, REC sub-folders, a Notes one, plus various WG-driven sub-folders.
268 Changing backwards this structures is probably unfeasible and will lead to broken links in existing references globally, but best practices and guidelines should be considered to give a simpler and more robust tree structure to the IVOA documents repository, alongside helping IVOA Doc maintenance task.
270 \subsection{ivoa.info domain}
271 The second domain registered by IVOA is the \url{ivoa.info} one. Is it registered only to prevent someone else from registering it. Does it makes sense to use it for some service of the IVOA, such as a mirroring solution ?
273 \section{Changes from Previous Versions}
275 No previous versions yet.
276 % these would be subsections "Changes from v. WD-..."
277 % Use itemize environments.
279 \bibliography{ivoatex/ivoabib,webassets}
281 \end{document}


Name Value
svn:executable *

ViewVC Help
Powered by ViewVC 1.1.26