CGI Applications

Written by Sabu Francis

Tangential Solutions. India

This tutorial is written in a question and answer format. The entire tutorial will have to be read linearly. The tutorial has two sections: Basics and Advanced. The structure of the reading material is intuitive: As you read the tutorial, you would encounter new concepts/terminologies. These get explained in sub-sections of the tutorial as soon as possible.

Note that CGI support is available only in the Commercial Edition of Visual Prolog.

You should be familiar with basic programming in Visual Prolog Concepts of class, interface, objects, etc. are not taught here. Neither are Prolog concepts of backtracking, clauses, predicates, etc. taught. It is assumed that you are familiar with all those. The examples developed in this tutorial do NOT depend on the creation of objects of any class, so it should be easier for those who are not familiar with object-oriented languages.

The final example uses Javascript for client side processing. It uses the object oriented features of Javascript. The reader of this tutorial is assumed to be familiar with this aspect of Javascript, at least. A good place to pick-up the object-oriented features of Javascript is http://www.webreference.com/js

What is CGI?

CGI stands for Common Gateway Interface. It is a specification recommended by the World Wide Web Consortium for one kind of information streaming in and out of web servers.

As you maybe know, HTML browsers (such as Internet Explorer, Netscape etc.) send and receive information from web servers. But the experience of the world wide web is not a simple act of just reading some hyper-linked HTML files. There is interaction. Data moves both in and out of a browser, including data filled into forms in the browser by the user. There are many things the web server cannot handle on its own. How does it then handle such interactive data?

A web server is a primitive creature actually. It usually serves out HTML files; and other files of mime-types that it recognizes, back to the browsers. It usually cannot do advanced stuff, like accessing a database in the background and sending database queries back to the browsers. CGI was invented to increase the functionality of web servers. It allows web servers to talk to CGI applications residing on the same machine as the web server itself. CGI applications do the job that the web server by itself is unable to do. For example; accessing databases in the background or conducting sophisticated searches. Web servers that use CGI, are usually able to send more information to the browsers than those that do not use CGI.

What is the significance of a 'common gateway?'

The word 'common' signifies that the convention is accepted by all web servers. The word 'gateway' represents the fact that data is obtained, as if by stepping through a 'door' or a gateway into other processes that lay on the other side. Metaphorically, a web server stands on one side of the gateway and a CGI application stand on the other side and both of them talk to each other using CGI.

What is streaming?

By the term streaming, we mean that the information is processed one character at a time in a long queue. Think of a pipe whose diameter allows only one character to pass through at any instance. The characters passing through the pipe will emerge at the other pipe end, only one character at a time, in the form of a stream.

Just like in a pipe, where water cannot go streaming in both directions; a stream in the programming world cannot be both readable and writable simultaneously. Either characters come out from the stream (which means the stream is a readable stream), or characters are placed into the stream (a writable stream) but the same stream cannot be used for both purposes by a single program.

In the case of CGI applications, the characters are in ANSI format, and so they are 8-bit characters. This point has to be noted later on, as Visual Prolog by default uses Unicode characters, which are 16-bit. There are predicates available in Visual Prolog that collect streamed ANSI characters together, so processing streams would not be a problem. Eventually, it must be pointed out, our experience of the world wide web happens because of the streaming characters arriving through the Internet pipeline into our browsers one character at a time. We just don't notice it though, because once the individual characters reach the browser, they are quickly assembled together.

What is a mime-type?

Think of mime-type as a file extension. It tells the web server the kind of files it and its clients (i.e., browsers) can handle. Most web servers determine the mime-type by the file extensions it sees. When a browser interacts with a web server, it initially informs the web server about the mime-type handling capabilities the browser has, along with the resource (file of a particular mime-type, etc.) it wants the web server to give. The web server sends over the requisite file to the browser for the browser to display. Examples are: .html, .jpg, .gif, etc. Non-recognized mime-types are either not sent to the browser by the web server, or are sent as plain ASCII or as HTML (the actual behavior depends on the web server).

As the information is sent as a stream to the browser, the web server embeds some headers at the start of the stream, containing the mime-type. See Illustration 1 for a typical example of what gets sent across from a web server to a browser. Once these files with mime-types reach the browser, it analyzes the headers, deduces the mime-type the data represents, and as per its own internal table of mime-types it can handle, it displays the data accordingly.

Illustration 1

What is a header?

A header, in the context of this tutorial, is a line of text followed by a carriage return embedded in the stream before the actual data begins. In Illustration 1, the lines you see above the data, are the headers that were passed by the web server to the browser. There is a blank line separating the headers from the actual data.

Notice the 'Content-type:...' header in the illustration. That line indicates the mime-type of the data. This particular line is specifically relevant to CGI applications, as when a web server calls on the services of a CGI application to provide data to be passed on to a browser, it is clueless on the mime-type of the data that the CGI application would be giving. Hence, by convention, the CGI application is responsible for pass this particular header before the rest of the data (separated by a blank line). Note this point carefully when we actually get around to programming CGI applications, because forgetting either the Content-type header or the blank line (between the set of headers and the actual data), or both is a common occurrence and it can lead to a lot of agony.

What is a CGI application?

A CGI application or a gateway program (see Illustration 2) is an executable file (usually an .exe file), which does not receive any information via the keyboard or the mouse, i.e., it is not an interactive application (a GUI application would be an interactive application). Such applications are known as console applications. They receive their inputs using the stdin stream and they put out their outputs into the stdout stream.
The rest of the headers are added by the web server after a CGI application does its work, and the compiled information is then passed to the browser.

Illustration 2

In case of CGI applications, once you start them, they terminate on its own without any inputs from you. You can say that they work automatically like the old DOS command line programs such as xcopy, or format, etc.

A CGI application, however, is not just any console application. It would expect the inputs as per the CGI specification and its outputs should also conform to the CGI specification.

In certain situations, a CGI application can be written as a .DLL file; but such variations are web server dependent. There are even more exotic variations possible. For example, even batch files (.bat files) can be made into simple CGI applications, provided the web server is configured to accept such files as CGI applications. Let us not get into those variations. For the purpose of this tutorial we'll assume that a CGI application is an executable, console application (.exe file), adhering to the CGI specifications.

What are stdin and stdout streams?

stdin is the name given in various programming languages, to a special stream that acts as the input stream from the underlying OS (Operating System). Programs (such as those written in Visual Prolog) can read the stdin stream, and use the characters given by that stream as their input. Similarly, stdout is a stream managed by the OS into which programs will output their characters into.

The stdout and stdin streams can be daisy chained (i.e joined end to end) from different programs. For example, a program 'A' can write into its stdout stream, and then call a program 'B', which in turn will use the stdin stream to pick up what was written by the program 'A'. This is essentially the transfer mechanism used by web servers: in the previous example, the program 'A' represents the web server and the program 'B' is the CGI application. The stdin and stdout streams are often referred together as the console.

How does the flow of information take place, using CGI?

The information flow starts from a request placed by the client (i.e., the browser) with the web server. As noted before, the client tells the web server about the various capabilities it has, and also speciies the information that it requires from the web server. See Illustration 3, which shows what gets sent by a browser (the client) to the web server.

Illustration 3

The first line contains the method field. This indicates the desired HTTP method, the resource being requested, the protocol version, etc., i.e., GET /Tests/file.html HTTP/1.0. Though /Test/file.html looks like a path, it is actually known as a URL (Uniform Resource Locator) and it represents a whole lot of things; including actual filenames and directories on the machine where the web server is located.

Other common methods are POST (for sending data from the client to the server (not as part of the URL)) and HEAD (retrieve only header information about the resource).Other fields contain information about the client. The Accept: fields tell the server the different types of data that the client can accept. These are sent as MIME Content-type messages. Thus Accept: text/plain means that the client can accept plain text files, and so on. This MIME type is important, as this is how web servers tell clients the type of data being sent. The other headers are informational: User-Agent: - a program making the request, From: - who is making the request, Referer: - gives the URL of the document making this request, and so on. The server responds by returning the desired data. It first sends server response headers, which communicate information about the state of the transaction, the type of data being sent, and any additional information. This is followed by a blank line and then the actual data being sent to the client. We saw this, in Illustration 1. Where does the CGI come into this?, - you may wonder. Well, let us re-visit the requested resource, which is in the above example /Tests/file.html, if instead of that, it was /cgi-bin/helloworld.exe. Many web servers are configured to let the partial URL /cgi-bin/ indicate that helloword.exe is a CGI program, which the WEB server must invoke.

Hence helloword.exe would be called by the web server and it would pass on some information to helloworld.exe using some environment variables and the stdout stream (which is the stdin stream for helloworld.exe). The web server will then wait till the CGI application, helloworld.exe, finishes its work.

Once helloworld.exe ceases execution, the web server would gather the data written by helloworld.exe (including the Content-type header), and send it to the waiting browser.

Phew! All that should ideally happen in the smallest period possible.

Thankfully, all web servers are designed to do its job well and so we do not have to worry about the assembly work performed by a web server after it invokes a CGI application. Let us concentrate instead on how a CGI application should read information from the web server and then pass its data back to the web server, after doing its work.

What is an environment variable?

All OS (Operating Systems) allow programmers to insert name=value relationships in what are known as environment variables, somewhere in the computer's RAM (Random Access Memory). For example; the PATH setting in DOS is an environment variable. You can examine the various environment variables that are set for DOS (or Windows XP or Windows 2000 for that matter) using the SET command. Just give SET command on a command line prompt, and the console will display all the currently available environment variables. (In the displayed list, you would see PATH also).

The best part of the concept of environment variables is that you can cook up your very own variables and assign values to them. For example; if you want to keep a password around somewhere, you could do this using the SET command at the OS command line:

SET password=xyze

Though the value is a string, you don't have to give quotes. All values are internally stored as strings, and they are retrieved by Visual Prolog programs as strings. The significance of environment variables in CGI applications is that a web server uses various environment variables also to pass information to the CGI application that it invokes. Here is an example.

We learnt that a web server places a lot of data on the stdout stream before calling a CGI application. How does the CGI application know the number of characters to read from the flowing stream? To facilitate this, the web server places the content length into an environment variable called, CONTENT_LENGTH, and using that environment variable's value the CGI application can read the right amount of characters from the stream. Let us examine some Visual Prolog code from cgi.pro (as part of the CGI pack that came with Visual Prolog) that does this:

clauses
    retrieve_POST_string() = Result :- % Get the length
        LenStr = environment::getVariable('CONTENT_LENGTH'),
        Len = toTerm(LenStr),
        %Read the stream to the desired length only
        Stream = console::getConsoleInputStream(),
        Stream:setMode(stream::ansi(ansi())),
        %The stream is in ANSI mode!
        Result = Stream:readString(Len).

Apart from CONTENT_LENGTH, most web servers set the following standard environment variables before starting a CGI application. A few additions and subtractions may be possible to these environment variables, depending on the web server that you use. The CGI application, in turn can then look for those variables and use the values that are contained therein. The reader is requested to read about these environment variables elsewhere, else this tutorial would become too lengthy. Of these, the environment variable HTTP_REFERER can be useful in implementing some simple security measures (described later in Section B of the tutorial).

PATH_INFO
PATH_TRANSLATED
REMOTE_HOST
REMOTE_ADDR
GATEWAY_INTERFACE
SCRIPT_NAME
REQUEST_METHOD
HTTP_ACCEPT
HTTP_ACCEPT_CHARSET
HTTP_ACCEPT_ENCODING
HTTP_ACCEPT_LANGUAGE
HTTP_FROM HTTP_HOST
HTTP_REFERER
HTTP_USER_AGENT
HTTP_COOKIE
QUERY_STRING
SERVER_SOFTWARE
SERVER_NAME
SERVER_PROTOCOL
SERVER_PORT
CONTENT_TYPE
CONTENT_LENGTH
USER_NAME
USER_PASSWORD
AUTH_TYPE

How do I test CGI applications?

You would need to have a web server installed, along with your CGI application in order to test it. You would also need support HTML files, which will trigger the CGI application through the web server. They will obviously vary from one application to another. You'll become aware of such support files, when you study the examples.

As indicated somewhere before, a CGI application has to be placed in the same machine, where the web server is itself running. For reasons of security, a web server is able to fetch files from only a specific set of directories (the set is known as the web-path) on the host machine. Therefore, you should place a CGI application files (CGI executable file and HTML files) into one of these directories.If that was not the case, some malicious user may try to fetch files that should not have been retrievable.

Similarly, a web server can only invoke a CGI application if it is placed in the right directory. You cannot place the CGI application anywhere on the web-path (well, a few web server do allow that too, but not all). Now, all web servers have a configuration method (a separate configuration file, or an .ini file or the windows registry, etc.) to configure its various settings. One of the crucial settings is the location of the CGI applications. When you zero in on a web server for your CGI application, the first thing you should study is how that particular web server configures its CGI application directories.

Which web server to use?

As indicated before, web servers are not complicated pieces of software. There are many web servers available free of cost. A large list can be seen at: http://www.serverwatch.com/stypes/index.php/d2Vi . That list includes both commercial software as well as free ones. The complexity of the web server is usually the configuration of the web-path and CGI application directory/directories of the web server.

The TCP-IP protocol should be installed in the computer where the web server is to be setup (It usually would be, if you are able to browse the Internet from that computer) Once you have successfully installed a web server, you should be able to get data from the web server by supplying either the IP address or the domain name of the machine where the web server is installed. Please note that the CGI programs should work fine with the most of web servers including the IIS5 web server which is usually included with Windows 2000 by Microsoft.

Where to place the CGI application?

Ensure that the CGI application is placed correctly in such a web server's appropriate script directory. This will vary from a web server to a web server. In this tutorial, we will assume that you have placed it in such a directory that it can be invoked using a browser on the server machine with the following URL: http://localhost/cgi-bin/<APPNAME>.

For example, if you have a CGI application called example1.exe, then it should be accessible with the following URL, on the server machine: http://localhost/cgi-bin/example1.exe.

Can you show me a very simple CGI application created using Visual Prolog?

Yes, indeed! You should unzip the file example1.zip that is included into cgitutorial.zip into a convenient place and examine the Visual Prolog files therein. But, you can create such an application from scratch too by following the instructions given below.

Start a new Visual Prolog project. Note that the UI Strategy is Console:

The main Project tree would get displayed. It will include all the necessary files that the IDE has intelligently decided you may require.

Now, click on the main Build menu, and in that menu, select Build or press Ctrl-Shift-B. You'll notice something magical, as the build process continues. As the project is being built, the IDE will pull in all the relevant PFC (Prolog Foundation Classes) files that were not visible earlier in the Project Tree. After the build process is over, the Project Tree will now look like this:

Congratulations! You have finished 95% of the job. Now you would be having a console application in the EXE folder of your project, which currently does nothing. We need now to add the relevant code to transform the application into a CGI application

As explained in the earlier sections of this tutorial, a CGI application expects some input from the stdin stream (Along with some environment variables), then it does its work and sends out the output through the stdout stream. As this is a very simple CGI application (our very first one) we'll ignore all input arriving into our CGI application from the stdin stream, (which the web server may be sending to the CGI application). Instead, we'll make our little application send some output.

Double click on the file main.pro in the Project Tree, and go towards the end of the code:

You'll notice that the main goal of the program invokes a predicate called run from the PFC module mainEXE. That module in turns invokes the predicate called run() from main.pro, so let us have a look at that:

clauses
    run():-
console::init(),
succeed().  % place your own code here

We'll modify that code to make it look like this:

clauses
    run():-
        console::init(), %Line 1
        cgi::init(stream::ansi(core::ansi)), %Line 2
        stdIO::write("Content-type:   text/html\n"), %Line 3
        stdIO::write("\n"), %Line 4
        stdIO::write("<html><body><h1>Hello World!</h1></body></html>"). %Line 5

Let us now go through the code, line by line:

• Line 1: is a Visual Prolog requirement. All work using the console, should first start with console::init() .

• Line 2: captures the stdout stream object to CGI output. As explained in the paragraph of this tutorial, the CGI application is expected to send its output to the stdout.

• Line 3: Now the CGI application is prepared to send the first line back to the calling web server. As explained in the paragraph of this tutorial, the data has to be preceded by some headers. The header which is mandatory is the one that sets the mime-type of the data. Here, it states that the Content-type of the data is of the mime-type text/html. Notice that the line ends in a carriage return (\n).

• Line 4: To separate the headers from the rest of the data a blank line is needed. It is mandatory as per the CGI specification. So the CGI application now sends a blank line (\n) there. Forgetting to do this is very common in CGI applications (Read the paragraph of this tutorial for more), and it can lead to the malfunctioning of the application.

• Line 5: In the end the actual data is written out to the stdout. In this example, it is simply creating an HTML file and sending that. The HTML file contains just two words : Hello World!.

How to test Example1.exe?

Place it in the script directory of your web server, as explained earlier. Then access the following URL on the same machine as the web server http://localhost/cgi-bin/example1.exe using a suitable browser.

What is the functionality of such a primitive application?

The Hello World CGI application (Example1) is not actually as primitive as it seems. It is simple alright, but it can serve as the seed for much sophisticated stuff. Take the last line (Line 6 explained in Paragraph of this tutorial). There the complete HTML is created and then dispatched over the stdout stream to the web server, who in turn, sends it to the requesting browser. If you introduce a database access at that point, and add some more predicates for conditional processing of the data from the background databases, you can easily build a content management system

What about inputs?

As indicated earlier, we have ignored whatever inputs the web server may have given to our little application. If the CGI application needed some inputs, then the stdin stream would have to be read, and processed for information. Also, the environment variables set at the time of the CGI application operation would also have to be examined. This can be a little complex. Luckily for us, we do not have to do much programming in Visual Prolog when developing CGI applications. For example, to know what data the CGI application receives from the web server, all we have to do is to use the cgi::getString() predicate. (We have to include the cgi.pack from the PFC into our project). And if we need the data in a more convenient form, then we can use the cgi::getParamList() predicate. The individual pieces of data are assembled by cgi::getParamList() into name, value pairs similar to environment variables.

How exactly does information flow from an HTML document to a CGI application?

In the Section A, Illustration 3 we showed the internal details of the stream that flows from the browser requesting information from the web server. Much of the experience of the World Wide Web consists of the browser dispatching such requests. The web server in turn processes the request and sends the desired mime-type data back to the browser. The matter does not end there. In HTML, one can embed special interactive forms using the HTML FORM element, and within that FORM element, one can use interactive elements such as INPUT, TEXTAREA, etc. The data entered into such forms can be dispatched to the web server. In the form it is specified:

• which action the web server should take with the form information;

• which method to be used to carry out the action.

Illustration 4

As seen in the illustration, the ACTION statement is nothing but a URL specifying the CGI application that is to be invoked by the web server. In the example given in the Illustration, it is a PERL application called prgm.pl. Data is passed to prgm.pl by the web server using the GET method.

What is the "GET" method?

In a GET method, the entire data passed to the CGI application is assembled into the URL when it is sent from the browser to the web server. Suppose you wanted to send three variables (firstname, lastname, and email) along with their values to the CGI application called myemailer.exe, then the URL would read as:

http://localhost/cgi-bin/myemailer.exe?firstname=sabu&lastname=francis&email=sabu@somewhere.com

When assembling such URLs, invalid characters are to be first converted to their URL encoded versions. For example; if any of the values contain a space, it would have to be written as %20, and a colon (:) will have to be written as %3A. This form of coding is called URL encoding. The name value pairs use the "=" sign as the separator between the variable name and its value. And to separate one name value pair from another, the ampersand (&) is used. Once the URL reaches the web server, the CGI application is triggered (myemailer.exe in this example) and the entire list of name value pairs are passed along to the CGI application via an environment variable called QUERY_STRING (See paragraph A.3.1 of this tutorial for a list of environment variables that are used by web servers while communicating to CGI application) The value of QUERY_STRING would be everything that appears after the "?" In the above example, it would be:

firstname=sabu&lastname=francis&email=sabu@somewhere.com

What is the "POST" method?

In a POST method, values are passed to the web server using the various interactive HTML elements that constitute the valid sub-elements of the FORM element in HTML (such as INPUT, TEXTAREA, etc.) These values are taken in by the web server, and passed onto the specified CGI application using the stdout stream. The CGI application, in turn, reads its stdin stream to retrieve these variables and the values they contain. (As explained in the paragraph A.2.1 of this tutorial, the stdout stream of the web server is readable as the stdin stream of the CGI application that is called by the web server) Please note that even in the POST method, it is possible to define a QUERY_STRING. For example, you can specify some parameters after giving a "?" at the end of the application name, just like in example shown in paragraph B.1.1. So, the CGI application can read data both from the stdin, as well as from the QUERY_STRING environment variable. This capability is used by our final example (in Example3.zip that is included into cgitutorial.zip) to differentiate between various types of POST processing.

What are the advantages and disadvantages of the GET and POST methods?

The GET method is bookmark-able. That means, the entire URL contains all the data that is needed to be sent to the CGI application. The disadvantage is that there is an upper limit (of approx. 1000 characters) that can be sent in this manner. Another disadvantage is that whatever data that is being sent to the CGI application is observable during the transit.

The advantage of POST method is that it can handle a large volume of information. Also, the data is not examinable easily in transit (other than by using snooping tools) And if you are using a secure server, and the browser has connected to the web server using a secure connection, then even snooping tools would probably not reveal the data that is being transmitted across to the web server. The disadvantage of the POST method is that you would usually need to write an HTML FORM element in the HTML, and populate that FORM with the correct HTML interactive elements such as INPUT, TEXTAREA, etc. for the POST method to work properly. There are ways to avoid a FORM element directly, but that involves clumsy Javascript programming within the HTML.

What is URL Encoding?

a) Non-ASCII characters (codes >128) are encoded via their URL octal encodings: %xx

b) Disallowed or special ASCII characters are encoded via their URL encodings. The disallowed or special ASCII characters are: ‘~!#$%^&()+={}|[]\:";´<>?,/ TAB That is: every punctuation character except the five characters: @ * _ - and . (the dot)

c) Space characters are encoded as plus signs (+) or as %20

Does the information transfer take place from the web server to the browser all in one stream?

Not always. Only the bare minimum HTML file, like the one shown in Illustration 4 is sent from the web server to the browser in a single stream. Most HTML files have other mime-types embedded within, for example, images or Javascript code. HTML elements such as <SCRIPT SRC="...">, <IMG SRC="..."> trigger individual connections to the web server. For displaying or using each of these separate mime-types, the browser establishes separate connections with the web server. That is the reason, why one finds that the browser builds up parts of the page separately (like the images), in a checkered fashion. We can write separate CGI applications for each of these streams, if so required. For example, we can write a separate CGI application for sending Javascript code to the browser, and another one for sending an image. This will be useful when we examine the final example (the threaded discussion board), later on in this tutorial.

But do we need to write so many CGI applications for handling the various streams required in an HTML file?

You are right! One does not really have to write separate CGI applications for each of the streams that are needed in one web session in the browser. There are ways around it. In the final example, (Example3.zip), we will show how to develop a CGI application that uses the QUERY_STRING to distinguish the various kinds of streams that are to be sent to the browser. The run() predicate that is called at the beginning examines the QUERY_STRING and diverts the execution of the code depending on the value passed in it. Thus, even though we have written only one executable file, it is like having 5 different CGI applications because the behavior changes as per the command given in the QUERY_STRING.

What are the advantages and disadvantages of CGI applications?

CGI applications extend the functionality of web servers almost without limits. The only real disadvantage is that CGI applications need to do its work very, very fast. If it takes too much time to process data and send its output, the web server will timeout the application and ignore any output that the CGI application may supply after the timeout.

Are there alternatives to CGI applications?

There have been attempts to change the specification, such as using pipes and sockets. Advanced web servers have built-in API (Application Programming Interfaces) that allow programmatic extensions of the web server itself to handle CGI traffic. These extensions are written as DLL files usually. All of them were done for achieving greater speed. However, they raise problems of compatibility.

Fortunately, the core CGI specification is still adhered to practically by all web servers. So, if you resist the temptation to explore variations of CGI and stick to the main specifications, you would be able to produce CGI applications that can be used on a wide variety of web servers.

How do I speed up CGI applications?

The examples shown in Example1.zip and Example2.zip that are included into cgitutorial.zip do all the processing of the data at the web server end. This includes, rendering information of the data. By rendering information, we mean all the HTML information with which the browser can actually show it on the window at the client end. Assembling the complete HTML at the web server end itself often is time consuming. Also, there is an added disadvantage: the look and feel of the assembled HTML cannot be easily changed without recompiling the CGI application itself. One very simple way to avoid this disadvantage is to send the bare bones data back to the browser in the form of Javascript, and let the browser do all the actual rendering of the information in the form of the final HTML that gets displayed.

Can CGI applications developed in Visual Prolog, help in client side processing?

In such a strategy, where client side processing is used, we can use Javascript to help us. Javascript is nowadays a very stable, object oriented language. A lesser known fact is that Javascript and Visual Prolog domains share practically the same syntax and this can be exploited to perform client side processing.

What is the connection between Visual Prolog domains and Javascript objects?

One wonderful aspect of Visual Prolog is that practically all domains in Visual Prolog can be reflected (i.e. recreated) in Javascript. Even those that represent nested structures can be recreated. The only exceptions are predicate domains. This unique connection between Visual Prolog and Javascript can be easily exploited, as seen in the following example:

Say, there is a domain defined thus in Visual Prolog:

nameStr= nm(string NameOfPoster, string Email).

The same domain can be represented in Javascript using an object. Here is its constructor, and a function which returns objects created using that constructor:

//Note: The following is a Javascript file named jsobjs.js.

// the constructor function nm1( NameOfPoster, Email) {

   this.name=NameOfPoster;
   this.email=Email; alert("Created a JS nm1 Object!");

}

//a function that creates an object using the above constructor function nm(NameOfPoster,Email) {

   return new nm1(NameOfPoster,Email);

}

The above functions are to be written into a file called jsobjs.js. The above strategy (one constructor + one function) would be used for each domain you define in Visual Prolog. Let us examine the reason for this approach.

Suppose, you have a Visual Prolog predicate that needs to send a bound variable of the domain nameStr to a Javascript stream, how would we do it?

class predicates
    returnNameStr : ().
clauses
    returnNameStr() :-
        V=nm("sabu","sabu@somewhere.com"), %Line 1
        writeNameStr(V).  %Line 2
 
class predicates
    writeNameStr : (nameStr TheName).
clauses
    writeNameStr(V):-
        stdio::write("Content-type: text/javascript \n\n"), %Line 3
        stdio::write("V="),  %Line 4
        stdio::write(V).   %Line 5

Let us assume that you have written a CGI application called "myjs.exe" where the above basic code is integrated .The source code of such an application is not fully given here. It is left as an exercise for you. (Hint: call the predicate returnNameStr() in the run() clause body, in a console application. See Example1).

Now, let us write a small HTML file as shown below:

<html> <script src="jsobjs.js"></script> <script src="/cgi-bin/myjs.exe"></script> <body> </body> </html>

You would notice that even though the HTML file is actually a blank one; when you load that HTML file in a browser (using the HTTP protocol), it will show an alert box like this:

Thus, we have successfully transferred a Visual Prolog domain to Javascript, as a Javascript object. Instead of displaying a pop-up alert like the one shown here, you can ask Javascript to actually render the contents to you in a more pleasing manner.

How does this little CGI Application work?

The main action starts with the returnNameStr(). In Line 1, a variable V is created in the domain nameStr. Then in Line 2, that variable is written out through stdout, using another predicate, writeNameStr(). Let us see how that predicate works. Line 3 is the header line (with a blank line at the end) that is required to send the mime-type to the calling browser. But wait a sec! In this example, unlike all our previous examples, the mime-type is text/javascript and not text/html. The reason is that unlike earlier examples which were outputting HTML, this CGI application is giving out Javascript.

Line 4 sends out just the string: "V=" which means the Javascript being written is defining a variable called "V" (Note this is a Javascript global variable, and it is a coincidence that we've chosen to name it as V, like the Visual Prolog variable in returnNameStr(). )

Line 5 then writes out the Visual Prolog variable V, as a string. This is where all the magic happens: The internal domain to string converter in Visual Prolog ensures that the string representation of the bound variable of the namestr domain comes out exactly the way you specified for the function nm(...) in Javascript!. Now, let us turn our attention to the HTML file that uses the CGI application (myjs.exe) The first <SCRIPT> statement calls the previously written jsobjs.js Javascript file which contained the Javascript object that reflected our Visual Prolog domain. Once jsobjs.js is interpreted by the browser, it gets the capability to internally create Javascript objects called nm1 (Note that the Javascript object is called nm1 and not nm. nm is the function whereas, nm1 is the class ) The second <SCRIPT> statement in that HTML file, calls this CGI application using the SRC parameter. This script supplies the data to the browser, for it to act upon. Effectively you can replace the second SCRIPT statement with the following SCRIPT block in the HTML file:

<script> V = nm("sabu","sabu@somewhere.com") </script>

...because that is what that got sent over by our CGI application. Note that the functor nm that Visual Prolog uses to distinguish the namestr domain, is used cleverly to invoke the function nm(...) in Javascript. At this point the nm(...) function, creates an object of the class nm1 and assigns it to the Javascript global variable V . If you now take a look at the constructor for nm1, in the file jsobjs.js, it calls an alert dialog whenever an object of that class is created. Finally, this example is a good example of client-side processing that we had set out to do. If you notice, the CGI application practically does nothing but dispatch a bound variable of a particular domain over to the browser, and the rest of the activity is actually performed at the browser's end, where the same bound variable reappears as a Javascript object.

Can you show me an advanced CGI application in Visual Prolog that uses Javascript objects?

This is demonstrated in the last example of this tutorial, Example3.zip that is included into cgitutorial.zip contains an advanced Visual Prolog 6.0 project called Discboard. After extracting the contents of example3.zip to a suitable folder, load discboard.prj6 into the Visual Prolog VDE and examine its source code. This is a full fledged working example that gives you a powerful and easy to use threaded discussion board.

Unlike the explanations that were provided for Example1.zip and Example2.zip, Example3.zip is better explained using the comments left in the source code along with the explanation given here.

Understanding the domains needed

In most advanced applications, one of the first things to be done is to develop a suitable data-structure which addresses the problem at hand. The discussion board that is being modeled here is similar to the one developed by Matt Wright (See http://www.scriptarchive.com/demos/wwwboard/wwwboard.html for a demo) Let us therefore understand the concept of a threaded discussion board: At first glance, it looks as if it contains a list of posts. By the way, a post is a submission made by someone into the discussion board. However, on closer scrutiny, you'll notice that it is actually a tree structure: Each post can have multiple replies which appear as its children. There are no limits to the number of replies, and the number of posts.

A new post is technically known as a thread. It can spin-off a separate tree of its own as it accumulates posts as its children (when people reply to it, and its children, etc, ad infinitum). A list of such separate tree-shaped threads becomes the entire discussion board.

If you study the domains declared in the file discboard.cl, the thrd domain captures the essence of a threaded discussion board. If you study the last parameter of that domain, which is a thrdList you will notice that it is in fact a list of the thrd domain itself. Thus, the entire tree structure of each of the main posts of the discussion board is in that domain.

The rest of the arguments of the thrd domain are self-explanatory when you study discboard.cl: They represent the rest of the issues concerning a single post, such as the name of the person making the post, his/her email address, the date and time of the post and the IP address of the person making the post.

Now that you've understood the domains, let us take a look at how it executes

The main execution of the CGI application starts from a predicate called execute(), which is towards the end of the module discboard.pro. The first thing it does is to examine the QUERY_STRING environment variable, and then redirects the execution of the program to one of the five clauses of a predicate called exec(). Depending on what was sent with the QUERY_STRING, the program behaves in a different way.

Therefore, for all practical purposes, this one CGI application acts as 5 different applications. The QUERY_STRING commands given could be one of these five: addpost, replypost, delpost, showtree and showpost. (The QUERY_STRING also passes along the name of the database file that is to be used for the threaded discussion). Of these 5 functionalities, showtree is the one that utilizes Javascript objects, and sends them across to the browser using the text/javascript mime-type.

Rest of them are quite intuitive in functionality and perform the required functions needed to maintain the data of a discussion board. For example, when a new post is created, then a thrd term is inserted into a specific chain ("__THREADS__") of the database of the discussion board. When the post is reply to an earlier post, then it is inserted as a child to a previously existing thrd term in the database. All those procedures, return HTML data back to the browser, showing the status of the operation. For your convenience, each of the four functionalities (addpost, replypost, delpost, and showpost) is written in its own section of discboard.pro, demarcated by its own class predicates. While reading the code in discboard.pro, concentrate on one such block at a time, and you'll quickly grasp the four functionalities.

However, the showtree functionality requires some explanation.

When the showTree() predicate does its job, what it really does is that it fetches the list of top thrd posts from the database, and sends the same list across as a Javascript array of function calls. This array is received by the file showposts.html which is able to parse them correctly into an array of Javascript objects due to the Javascript code written in dboard.js

You would need to examine the Javascript code in dboard.js to see how a thrd(...) Javascript function creates a Javascript object called thrd1(...), and how an array of such functions ends up creating an array of thrd1(...) objects. The basic method by which a Javascript function ends up creating a Javascript object is the same as the one we saw in the paragraph B.7.1, where the Javascript function nm(...) created a Javascript object of class nm1(...) Once the array of Javascript thrd1 objects are in place, there is a Javascript function called wrtAll(...) defined in dboard.js which utilizes the information in the array to form the required HTML which eventually gets displayed in the browser. (Dboard.js and showposts.html are files present in the web folder).

After you understand the source code of this CGI application, then to understand the client-side processing part, you will have to study the files required at the client end:

These are: dconfig.js and dboard.js that are the Javascript files that control the client side processing. Of this dconfig.js is a simple configuration file, whose parameters can be changed to suit your discussion board. Dboard.js is the main Javascript file containing all the constructors and function definitions that correspond directly to the thrd domain and other domains defined in the Visual Prolog source code. The functions in Dboard.js translate the instructions received from the CGI application to the corresponding Javascript objects, of the manner described in the paragraph B.7 of this section of the tutorial.

Then you need the corresponding HTML files also: showposts.html, rep.html, postmsg.html, replymsg.html, delmsg.html. Finally, dboard.html is also needed, which assembles all the HTML files into a frameset. The files; postmsg.html, replymsg.html and delmsg.html contain the forms that are used to do the interactive work.

When you invoke the discussion board, the discussion board starts its work by the activities within showposts.html. That file may seem like an empty HTML file, but that is the file which invokes the showtree command of this CGI application, and brings in the array of Javascript functions that defines the Javascript objects, to create the main discussion board tree. Once the discussion board tree is correctly rendered into the browser frame, then it is ready to service the 'clicks' that it would receive. Clicking any of the message links displays the correct message in the bottom frame of the frameset. And the top frame (which uses the file rep.html)'' contains the rest of the interactive buttons and forms needed for the discussion board.

rep.html contains few forms, and a dispatching system to invoke the correct HTML form depending on which function was invoked. For example; the Add Post button will make the postmsg.html page to pop-up in a separate window. All the HTML and JS files are sufficiently commented so that they are self-explanatory on what each one does.

Security Issues

A few words of caution before we end this tutorial: CGI application do not have the same restrictions of the web server. A web server can work only within a contained area of the web-path. No other directories of the computer where the web server is installed, are accessible to a web server. If you keep your private data outside the web-path, you can be reasonably sure that it will not leak out to any prying browser. However, a CGI application has no such restriction. It gives it a lot of power, but if that power is not harnessed correctly, one can write CGI application that may unnecessarily expose large portions of your data to outside view.

One another problem with a CGI application is that it is quite easy to write a CGI application that is oblivious of where it is running. Such CGI application can be easily hijacked by other people without your knowledge. For example, all the three CGI application developed in this tutorial are prone to hijacking. This can be illustrated by the following example: The following HTML hyperlink code:

<a href="/cgi-bin/example1.exe">First example</a>

can be used on the same web server where example1.exe was located (say a website call http://site1) If that HTML was found on site1, then clicking on that link would yield the "Hello World" result, as expected. Now let us take that HTML code and place it on another site, say http://site2 This time, we'll NOT copy example1.exe to the cgi-bin directory of site2. As expected, when you click on that link it will not work (as expected) because example1.exe is not really residing on site2. Now let us try to hijack example1.exe. We'll use the following HTML code:

<a href="http://site1/cgi-bin/example1.exe">

   First Example on another server</a>

Now when we click on the link, it would work properly even though the HTML code is residing on Site2 and Site2 does NOT have example1.exe anywhere on the machine!

How to prevent hijacking of CGI Applications?

One very effective way to prevent hijacking of CGI application is to examine the environment variable called HTTP_REFERER. The value of that variable contains the information regarding the HTML file which was referring to the CGI application at the point of execution. This can be easily parsed to see if the website is a valid one, and the CGI application can be made to proceed accordingly. Let us modify example1 (refer the paragraph A.5.1 of this tutorial) and see how we can prevent hijacking of that example.

clauses
    run():-
        console::init(), %Line 1
        cgi::init(stream::ansi(core::ansi)), %Line 2
        stdIO::write("Content-type: text/html\n"), %Line 3
        stdIO::write("\n"),%Line 5
        run2().
 
class predicates
   run2 : () .
clauses
   run2() :-
        checksOut,
        !,
        stdIO::write("<html><body><h1>Hello World!</h1></body></html>"). %Line 6a
 
    run2() :-
        stdIO::write("<html><body><h1>Access denied!</h1></body></html>"). %Line 6b
 
class predicates
   checksOut : () determ.
clauses
    checksOut() :-
        Refr = environment::getVariable("HTTP_REFERER"),
        string5x::concat("http://localhost", _, Refr),
        !.

To use the above example, you would have to include the string5x and cgi packs into the example1 project. (And while building, the VDE may prompt you to add more .packs to the project. You should click on "Yes to all" when that prompt appears) After you compile this modified example, you would notice that example1.exe would work ONLY when it is run via an HTML page that is displayed from the http://localhost url. The magic happens in the checksOut() predicate, which finds out what was the value of the environment variable HTTP_REFERER. This environment variable is passed on by the web server to the CGI application, and it contains the URL of the page which called the CGI application. The second line of the checksOut() predicate uses the concat predicate to see if the calling page was indeed from the web-address http://localhost You can check out the above example by trying to access the script using the URL http://127.0.0.1 (127.0.0.1 is the IP address for localhost), and you'll find that the script will say Access Denied!

Conclusion

CGI application give a powerful route to extend the functionality of a web server. It can come at a cost of speed, but if we use client side processing using Javascript, much of the processing can be handed over to the browser; leaving the CGI application free to do only the bare minimum work. CGI application are nothing much more than console applications that are specifically designed to do their activities very fast.