Web Services

This tutorial describes how to create a Web Application with a HTML/JavaScript front-end that run in the users browser and a backend written in Visual Prolog.

The emphasis is on the Visual Prolog back-end, which implement a JSON-RPC web service. The HTML/JavaScript code is mainly included to make the example self-contained and to illustrate how front-end code can interact with a Visual Prolog web service.

It is described how the service can run as a stand-alone HTTP/HTTPS server

Also notice that this tutorial will not consider SOAP at all.

The tutorial corresponds to the webService examples.

A web service can as described here be implemented as a stand alone server. Tt can also be implemented as an IISAPI plug-in, but we recommend using a standalone instead of using an IISAPI plug-in.

Overview

In this section we will try to describe the overall picture of the entire application and the way it works. What takes place is actually rather complex, but fortunately most of the complexity is handled automatically. An understanding of the overall process including knowledge of some of the complexity will however make it easier to understand what needs to be done and why.

The overall work flow of the example application is as follows: The user browses (in a web browser) to the page http://<server>/test.htm (or http://<server>/calendar.htm), and the server returns the corresponding file to the browser. The browser evaluates the contents of the file, which may result in additional requests for several cascading style scripts (css) files, JavaScript (js) files, images, etc. Furthermore, the HTML file contains embedded JavaScript code, which will make remote procedure calls (RPC) to the web server. The server will execute the corresponding procedures and return the result to the browser. Finally, the embedded JavaScript uses the returned data to update the HTML contents of the browser.

The communication between the client and the server is performed by means of the HTTP (or HTTPS) protocol. So on the server side we need an HTTP server, which can both return files and implement remote procedure calls. The program in jsonRpcService_httpApi runs the service as a stand alone HTTP server that both handles file requests and implements the remote procedure calls. The program in jsonRpcService_isApi implements the remote procedure calls as an ISAPI plug-in, the IIS (Microsoft Internet Information Services) is configured to handle the file requests and to use the ISAPI for the remote procedure calls.

The stand alone HTTP server is implemented by means of Microsoft's HTTP Server API, which can share ports and URL's with the Internet Information Services (IIS), such that http://<server>/file, etc can go to our stand alone server while http://<server>/<something> can be handled by the IIS.

JSON & JSON-RPC

The remote procedure calls are implemented in accordance with the JSON-RPC 2.0 Specification specification, which implies that data is encoded in JSON (JavaScript Object Notation) format.

JSON syntax is a sub-set of JavaScript (hence the name): When evaluating a valid JSON text it will become a corresponding JavaScript object. You should however always call a JSON parser instead of evaluating the text: evaluating text is a security risk, since the text could be "code" rather than "data".

It is not necessary to understand the JSON format in details, because on the client side you will work with JavaScript and on the server side you will work with an object/domain representation of JSON: parsing and writing is left to standard software.

When performing a JSON-RPC call the client sends a JSON request object, and the server will respond with a JSON response object, unless the request is a notification, in which case there is no response.

Request objects have four members:

• jsonrpc: A string specifying the version of the JSON-RPC protocol. MUST be exactly "2.0".
• method: A string containing the name of the method to be invoked.
• params: The method parameters (omitted if the method does not take parameters).
  • by-position: a JSON array, containing the values in the Server expected order.
  • by-name: a JSON object, with member names that match exactly, including case, to the method's expected parameters
• id: A client created string or number used to identify the request. The id field is omitted for notifications.

A response object contains three members:

• jsonrpc A String specifying the version of the JSON-RPC protocol. MUST be exactly "2.0".
• id The value submitted by the client.

The third member is one of these:

• result (implying that the call succeeded) The result of the procedure call
• error (implying that the call failed) An object describing the error which occurred

Client side

Example This code shows one way to make a procedure call on the client side:

function httpPOST(URL, method, params) {
    var ReqObj = { jsonrpc: "2.0", id: getId(), method: method, params: params };
    var Req = JSON.stringify(ReqObj);
    var xmlHttp = new XMLHttpRequest();
    xmlHttp.open("POST", URL, false);
    xmlHttp.setRequestHeader('Content-Type', 'application/json-rpc; charset=UTF-8');
    xmlHttp.send(Req);
    var resp = xmlHttp.responseText;
    var respObj = JSON.parse(resp);
    var error = respObj.error;
    if (error) {
        if (Data = error.data) {
            log("<pre>" + Data +"</pre>\n");
        }
        log("<p>error : "+ error.code +": " + error.message + "</p>\n");
    } else {
        // respObj.result contains the result
    }
}

In brief:

• Create a request object
• JSON "stringify" it
• POST it using an XMLHttpRequest object
• JSON parse the response text
• Check for "error" member

There are many other ways to make client side code, typically you will use some standard JavaScript library packages that handles errors and perform the calls asynchronously. calendar.htm uses such library code, but that is outside the scope of this tutorial.

Service Code

In Visual Prolog JSON is represented by the domain json::jsonValue:

domains
    jsonValue =
        n; % null
        f; % false
        t; % true
        r(real Number);
        s(string String);
        a(jsonValue* Array);
        o(jsonObject Object).

There are only 7 kinds of JSON values: null, false, true, a number, a string, an array and an object.

An array is a sequence of JSON values represented as a list in Visual Prolog.

A JSON object is represented by the jsonObject interface in Visual Prolog:

interface jsonObject supports mapM{string Key, jsonValue Value}
...
end interface jsonObject

The interface contains a number of convenience predicates/properties (which have been omitted above), but in essence a jsonObject is simply a map from Key's (names) to JSON values.

Example This code:

J = jsonObject::new(),
J:set_boolean("bbb", true),
J:set_string("sss", "The String"),
J:writeTo(stdio::outputStream),
stdio::nl,
jsonPrettyPrint::ppObject(stdio::outputStream, J)

will produce this output:

{"bbb":true,"sss":"The String"}
{
    "bbb" : true,
    "sss" : "The String"
}

writeTo produces a condensed JSON representation suitable interprocess communication, etc. jsonPrettyPrint can be used to pretty print the values (for example when debugging).

When creating an RPC service you will have to implement an object that supports the rpcService interface:

interface testService supports rpcService
end interface testService
 
class testService : testService
end class testService

In the implementation you will inherit from jsonRpcServiceSupport which will handle the parsing, writing, etc of JSON:

implement testService
    inherits jsonRpcServiceSupport
    open core, json
...
 
end implement testService

You will register your procedures in the constructor:

clauses
    new() :-
        setProc_name("hello", hello),
        setProc_name("calendar", calendar).

There are four kinds of procedures, with corresponding registration predicates:

• setProc_array for registering a procedure that uses positional arguments in an array
• setProc_name for registering a procedure that uses named arguments in an object
• setListener_array for registering a notification listener that uses positional arguments in an array
• setListener_name for registering a notification listener that uses named arguments in an object

In the example there are two procedures "hello" and "calendar" that both uses named arguments. The "hello" code looks like this:

constants
    testService_error : jsonRpcError::serverErrorCode = jsonRpcError::serverError_last+1.
    % grows up from "last", which is used for unspecified server error
 
predicates
    hello : jsonProc_name.
clauses
    hello(_Context, ArgMap) = o(Result) :-
        Result = jsonObject::new(),
        Username = ArgMap:get_string("username"),
        if "error" = Username then
            raise_serviceError(testService_error, "'error' is an invalid user name")
        elseif "userError" = Username then
            exception::raise_user("'%Name%' is an invalid user name", [namedValue("Name", string(Username))])
        elseif "definiteUserError" = Username then
            exception::raise_definiteUser("'%Name%' is an invalid user name", [namedValue("Name", string(Username))])
        else
            Result:set_string("token", string::concat("Hello ", Username))
        end if.

First the hello procedure creates a jsonObject for the Result. Any kind of JSON value are legal as result, but hello is designed to return a JSON object with a "token" member.

Next it fetches the arguments from ArgMap. It expects a string argument named "username".

hello uses the value of Username to illustrate exception handling and success.

The "error" case illustrates how you can raise "service errors", that is errors that are specifically designed for the RPC service. The "userError" and "definiteUserError" cases illustrates what will happen with other exceptions that occurs in the procedure call.

Notice that the client side code should be programmed to deal with errors. On the server side you may of course also handle specific exceptions as in any other program, and perhaps you also want to log all errors somewhere.

If Username is not one of the error cases, we map "token" to a string in the Result.

Example We recommend that your jsonProc's only deal with argument extraction and result construction, and that you keep the real functionally in a separate business layer as illustrated by this code:

predicates
    hello : jsonProc_name.
clauses
    hello(_Context, ArgMap) = o(Result) :-
        Result = jsonObject::new(),
        Username = ArgMap:get_string("username"),
        Token = businessLayer::hello(Username),  % actual functionality is in a separate business layer
        Result:set_string("token", Token).

Stand-alone HTTP server

As illustrated by the jsonRpcService_httpApi program the testService (supporting the rpcService) can be used in a stand-alone HTTP server. It does not require much code and it is quite simple to adjust the code to specific needs. Nevertheless the complexity is quite large.

The server is based on Microsoft's HTTP Server API which cooperates well with Microsoft Internet Information Services (IIS). In fact, it seems that IIS is built on top of the HTTP Server API (or at least that both are built on top of some common base).

The HTTP Server api operates with the following entities:

• Creator/Controller Process
• Server Session
• URL groups
• Request queues
• Worker Processes

The creator/controller process creates a server session, which defines URL groups and request queues. The worker processes attach to the request queues and process the requests they receive.

In the example program the creator/controller process is also the only worker process. And it is outside the scope of this tutorial to consider more complex scenarios.

The server session is the "owner"/"holder" of the URL groups and the request queues, which are the things we really need to consider.

An URL group defined by a set of URL patterns of the forms:

http://<host>:<port>/<relativeURI>

https://<host>:<port>/<relativeURI>

When a request arrives at the computer a matching mechanism will match it against the URL patterns in the active URL groups, the matching will either fail or determine an URL group that covers the URL.

Each URL group is attached to a request queue, and requests matched to a certain URL group is placed in the attached request queue. URL groups that are not attached to a request queue are inactive and does not participate in the matching.

Worker processes dequeue requests from the request queues and handle them. A request queue can have several URL groups attached and each URL group can have several URL patterns. A worker process dequeuing from a certain request queue will therefore receive requests that match some URL pattern in some URL group attached to that queue.

In the example server we create two request queues each having a single URL group attached, and each of these URL groups having one URL pattern. Effectively:

• reqQueue_rpc receives requests of the form http://localhost:5555/jsonrpc/<something>, and
• reqQueue_file receives requests of the form http://localhost:5555/<something>'

reqQueue_rpc is a requestQueue_rpc handles RPC requests by feeding them to our testService.

reqQueue_file is a requestQueue_file which uses fileMapper to map the requests to a file name and an associated ContentType, the requestQueue_file will then handle the transfer of the file.

Server configuration: netsh

The example runs smoothly because it uses port 5555 on localhost. Normally you will however want to run applications on port 80 for http and 443 for https, and that will cause some security problems.

If you just change the port number and uses a public hostname you server program will get an 'Access is denied' exception when trying to add the URL patterns to the URL groups.

If you run the server "as Administrator" you will not get the access violation, but it is absolutely not advisable to run the program as administrator, because then it also have rights to do many other harmful things.

To make the program run in normal modes you will have to reserve the URL patterns in question for the user that run the server program.

This is done using the netsh.exe (net shell) program. You will need to manipulate the http URL acl's. For a detailed description see Netsh commands for HTTP.

Example This will give an idea of what to do (in a cmd prompt):

>netsh http add urlacl http://www.something.com:80/ user=SMT\serviceuser

Notice that the cmd prompt be run "as Administrator" to make modifications. If you just write "netsh" it will go into a prompt mode where change to http mode and then just write http commands

>netsh
netsh>http
netsh http>add urlacl http://www.something.com:80/ user=SMT\serviceuser
netsh http>exit

It is possible to make the URL reservations from Visual Prolog, but in case of conflicts and problems it may be better to use the standard program netsh.

Notice that the example program is a console program. In an larger scale context it would seem better to create a genuine windows service. Visual Prolog has a template project for creating services, but that subject is outside the scope of this tutorial.

Existing GUI program

In the webRPC example directory there is also a jsonRpcService_httpGUI, which illustrated how a GUI program can act like a stand-alone Web RPC Service. You should notice that it is not recommended to let a real service program have a GUI, but it can be quite useful if you need to let other programs interoperate with the GUI program, i.e. in a normal user login session. It can also be used as a "quick and dirty" way to expose already existing GUI code as a Web Service (perhaps as a test before creating the real service).

Multi-threading

No matter how you run the web service (stand-alone with or without GUI, or as ISAPI) you should notice that the HTTP requests run in parallel in several threads. So it is important to ensure that the request handling's are thread safe. The requestQueue_rpc::synchronizer property can be used to synchronize the execution of the RPC's, the property is set to a predicate which will receive runnables, when the runnable is executed the actual RPC call is performed. The runnable's can for example be executed in a monitor ensuring that only one at the time is execute, or be put on a queue and executed sequentially from that or as jsonRpcService_httpGUI does be posted to the GUI queue and thus executed by the GUI thread.

Visual Prolog Client

This tutorial and the mentioned examples has focused on the Visual Prolog server side, using a web browser as client side. But sometimes a Visual Prolog program may need to access a web service as client. The example jsonRpcClient illustrates how this can be done. The method is quite similar to the one used in JavaScript:

• Create a jsonRpcRequest object, with method name and parameters
• JSON "stringify" it using asString
• POST it using an XMLHttpRequest object (an instance of xmlHTTP or xmlHTTP60)
• JSON parse the response text (using jsonObject::fromString)
• Check for "error" member

In the example program you see a delayCall, whose purpose is to allow the GUI to be updated, without this call the GUI would not update until after the RPC have completed. The call is however irrelevant for the RPC itself.

Skype HTTP/HTTPS Conflict

Skype (potentially) conflicts with HTTP/HTTPS servers, because Skype (by default) reserves port 80 (HTTP) and 443 (HTTPS).

As a result you cannot run an HTTP/HTTPS server when Skype runs (in default mode).

The solution is to turn off port 80/443 in Skype.

See this video: Handling port conflicts with Skype on Windows

HTTPS

Finally a set if brief notes about Secure connections (SSL/TLS).

HTTPS is the HTTP protocol used over a secure socket. Both the IIS and the stand alone HTTP Server can use HTTPS. To use HTTPS the server computer must have a certificate. It is possible to create a certificate yourself, but a self created certificate is "untrusted" will therefore cause security notifications. This is sufficient for testing and development purposes, but for real scale purposes you will have to acquire a trusted certificate.

In brief a trusted certificate have to be obtained from a trusted provider. The trust lies in three things:

• The provider is itself trusted
• The provider have knowledge about the "real identity" of the certificate holder (through normal purchase chains)
• The provider can revoke the certificate if it is misused somehow

Notice that HTTPS by default uses port 443 (and HTTP uses port 80), and that you have to write "https" in the places where you would otherwise write "http" (e.g. https://+:443/mypath)

For test purposes you can create self signed certificates using the power shell command "New-SelfSignedSertificate":

New-SelfSignedCertificate -DnsName "*.localhost.xx"" -CertStoreLocation "cert:\LocalMachine\My" -NotAfter (Get-Date).AddYears(10)

If run "Manage Computer Certificates" you will find it in "Personal\Certificates" and if you copy it to "Trusted Root Certificates\Certificates" (it must be both places) then you computer will trust it.

Double clicking the certificate and viewing the details will reveal the ThumbPrint which is the identifier of the certificate.

You will use this identifier to tie the certificate to the connection. This is done by adding an "sslcert" record with "netsh":

netsh http add sslcert hostnameport=demo.localhost.xx:443 certhash=19bde81e1b443b28fe75553cc52e0d8ba99978f1 appid={00112233-4455-6677-8899-AABBCCDDEEFF} certstorename=MY

The "certhash" identifies certificate (the ThumpPrint in the certificate details), the purpose of the "appid" is a unclear to me but it is in any case a GUID.

To make test with "real" server names like "demo.localhost.xx" you can make an 127.0.0.1 entry in the "hosts" file (C:\Windows\System32\Drivers\etc\hosts):

...
# localhost name resolution is handled within DNS itself.
#	127.0.0.1       localhost
#	::1             localhost
127.0.0.1 demo.localhost.xx

In JavaScript you can get the protocol like this:

// JSON RPC handling
const apiUrl = window.location.protocol + '//' + window.location.host + '/api'

For web sockets like this:

// webSocket handling
const wsProtocol = ('https:' == window.location.protocol ? 'wss:' : 'ws:')
const wsUrl = wsProtocol + '//' + window.location.host + '/ws/'

See also

• JSON
• JSON-RPC 2.0 Specification
• Microsoft's HTTP Server API