CEA Interface (Working Draft)

The Virtual Observatory (VO) is general term for a collection of federated resources that can be used to conduct astronomical research, education, and outreach. The International Virtual Observatory Alliance (IVOA) is a global collaboration of separately funded projects to develop standards and infrastructure that enable VO applications.

Syntax Notation Using XML Schema

Note that as this document represents a draft the cea: and ceab: namespaces in attached schema and instance documents have the string "rc<n>" appended where "rc" stands for release candidate and <n> is an integer representing the iteration number of the release candidate draft of the associated namespace.

1.

The concept of a Common Execution Architecture (hereafter CEA) was introduced in [CEA]. This document represents a formal publication of the interfaces to a service that provides the execution capability of applications that can be described by the CEA application model that is now defined in a separate standard [CEAAPP]. This document defined the Common Execution Connector SOAP web service that was described initially in [CEA], as well as a new [REST] interface, both of which are conformant with the IVOA Universal Worker Service [UWS] pattern. The fact that many various kinds of application can be exposed with a single web service definition and application model is an important feature of CEA, as it allows universal server and client side software facades to be created that hide the details of the web service implementation, which in turn simplifies the adoption into existing frameworks such as workflows.

2.

The rationale behind creating a Common Execution Architecture have been expounded in [CEA] and [UWS] in detail, so here only the most important features will be highlighted.

Asynchrony - the interaction between the client and the server must be asynchonous to allow for network timeouts/failures during the time that the application execution is occuring. This necessarily means that there is a not a simple call/response interaction between the client and the server, but rather a series of calls and responses make up the complete interaction to invoke an application and obtain the results.
Independence from a particular implementation technology - The design of the application definition and the description of the interface was intentionally kept at a relatively "high" level so that specific implementations could be created using particular technologies, but allowing essentially the same programatic interface (API) to be used on both the client and the server ends of the interaction. This document describes two such implementation technologies, SOAP web services and REST style web services.

2.1. Relationship to UWS

The Universal Worker Service [UWS] pattern was originally inspired by the early CEA implementation within the Astrogrid project, which provided a pattern for asynchronous interaction and data staging with remote services using a SOAP web service interface. The UWS pattern attempts to abstract these ideas slightly further so that they might be reapplied in other services, and as such exists only as a document, there are no concrete 'pure' UWS implementations. The UWS pattern has had a separate track of development, notably the addition of a REST profile, which in-turn is being fed-back into this CEA standard.

2.2. Interactions

The interaction between the CEA client and the CEA server follows a similar pattern for both the interfaces. There are some extra intermediate steps that may be called between these steps, but the steps listed below are the essential interactions that can be performed on either interface for the complete execution of a job and retrieval of the results.

Client requests that a new "job" be created passing a <tool> element that describes the application to run and the parameters to pass to it. The Client receives a job identifier in return that it can use to identify this particular job in subsequent calls.
Client requests that the job be started.
Client repeatedly inquires about the status of the job until it is informed that the job is finished.
Client retrieves the results from the job.

The job goes through several phases during this interaction. The first 4 phases that are listed are what occurs for normal successful execution.

PENDING: The initial phase when the client has just requested the job to be created, and the parameters are being validated.
QUEUED: The job has been validated and accepted, but no request to start the job has yet been received
EXECUTING: The job is running, after the signal to start has been given.
COMPLETED: The job has successfully completed.
ERROR: An error of some kind has occured
UNKNOWN: The job is in an unknown state - no clear error signal has been received, but nor has it completed within an expected time.
SUSPENDED: The job has been requested to be executed by the client, but the server has not yet started the job, perhaps because of lack of available resources. (cf QUEUED). The job will be automatically started when resources become available without client intervention.
HELD: The job has been executing, but the server has suspended the job. The job will not be restarted without client intervention.

3. Interfaces

There are two possible interfaces to a CEA server using either Simple Object Access Protocol [SOAP] or "Representational State Transfer" [REST] technologies. A CEA server must implement one of these interfaces and should ideally implement both so that a CEA client has a choice to use whichever is most convenient in the given situation. These two interfaces are described in more detail below.

3.1. Common Execution Connector SOAP

The SOAP interface that implements the UWS pattern in CEA is called the Common Execution Connector and is described by the WSDL contract in Appendix A. The CEA SOAP interface adds a "call-back" feature to the UWS pattern, that means that the client does not have to continually poll the server to obtain job status, but instead can register to be "called-back" when the job status has changed. In large grids this can be beneficial as it cuts down on network traffic and cpu cycles compared with polling. There are two such call backs that can be registered.

A progress listener, which can be used to receive messages about the current status of the job. This is registered with the registerProgressListener() operation of the CommonExecutionConnector port. The operation expects the endpoint of a server that implements the CEAJobMonitor WSDL contract (see Appendix C) which implements a one.
A results listener, which can receive the results when the job has completed. This is registered with the registerResultsListener() operation of the CommonExecutionConnector port. The operation expects the endpoint of a server that implements the CEAResultsListener WSDL contract (see Appendix D).

A typical interactaction with the CommonExecutionConnetor interface is illustrated in the following UML diagram.

CommonExecutionConnector UML sequence diagram

Each operation in the CommonExecutionConnector interface is described in more detail in the following sections.TBC

createJob

The job is specified by passing a tool document.

Note on output parameters. The output parameters need not usually be sent of course, as they are output parameters! However, to instruct the CEA server to automatically stage the output to somewhere the output parameter should be sent with the indirect attribute set to true and the value set to the URI of the location where it is desired that the output should be sent. If this is not done the CEA server will attempt to return results in-line in the getResults SOAP call.

registerProgressListener

This allows the CEA client to register the end-point of a service that progress log messages may be sent to. This does not stop a client also requesting similar messages by polling with a getPhase call. The frequency of messages that are sent is left as an implementation detail for the CEA server, but it would be useful to sent messages when for instance;

the execution phase changes.
a new result of the execution becomes available, in cases where some results may be available before completion of the entire job.

registerResultsListener

Allows the CEA client to register the end-point of a service that the results can be sent to when ready. The client may also request results explicitly by using the getResults call.

getResults

Request the results from an execution. These will be returned in a ceat:result-list-type structure, which may return results

in-line as the value of the result element
indirectly by pointing to an external URL. e.g. to a VOSpace location.

delete

Request that the job be deleted. If the job is currently running then it will be aborted and the record of the job deleted from the CEA server. If the job has already completed then this call will delete any record of the job from the CEA server and free any resources associated with the job.

getPhase

Returns the execution phase of the job, along with some associated metadata about the current status of the job.

getExecutionSummary

Returns a detailed summary of the job including the input parameters and the results.

setTerminationTime

Sets the time at which the CEA server will automatically delete the record of the job and also reclaim any resources associated with the job. After this time it is not possible any longer to retrieve any metadata about or results from the job. Any results that have been staged to another storage location will of course not be deleted.

3.2. UWS-PA REST

3.3.

4. Registering CEA Services

The registration of CEA components is covered in detail in [CEAAPP], however it is worth noting some specific points here about service endpoints.

the two different interfaces have their own types in VOCEA.xsd, and should be registered individually with these types in the <capability> of the resource.
the <accessURL> that is registered for the REST service is the one that starts at the path element above /(jobs)/ level of the URI interface hierarchy as described in [UWS], and that ./jobs is the concrete representation of the (jobs) level of the path in that specification.

5. Security

CEA implementations generally follow the security recommendations of the IVOA GWS group [SSO]. Specifically for the SOAP version of the interface, message level security using WS-Security is implemented, and for the REST version transport layer TLS security is used. At a minimum CEA servers should check client authentication via these means. A CEA server should publish it's authorization policy, but as a minimum policy the server should allow read and write access only to the creator of a job.

6. Possible Extensions to CEA Interfaces

There are some extensions to the CEA interface definitions that could be made, that would detract from the "purity" of the original design, but might make the interfaces more easily usable in many practical deployment scenarios.

Allow the original job to be created in the REST interface with a POST containing a standard post variables to represent the parameters rather than having to POST the <tool> document. There are some practical problems in representing whether parameters are indirect or not, and representing array valued parameters, but naming conventions could overcome these. It would probably also be beneficial (though even less RESTful) to allow the parameters to be encoded in the URL and use http GET to create the job - thus allowing easy "bookmarking" of a particular invocation request.
In the SOAP interface, expose each registered application as a single synchonously callable SOAP RPC style web service.