CGI Applications

CGI Applications
CGI Applications Basics Advanced

CGI stands for Common Gateway Interface. It is a specification recommended by the World Wide Web Consortium for one kind of information streaming in and out of web servers.

As you maybe know, HTML browsers (such as Internet Explorer, Netscape etc.) send and receive information from web servers. But the experience of the world wide web is not a simple act of just reading some hyper-linked HTML files. There is interaction. Data moves both in and out of a browser, including data filled into forms in the browser by the user. There are many things the web server cannot handle on its own. How does it then handle such interactive data?

A web server is a primitive creature actually. It usually serves out HTML files; and other files of mime-types that it recognizes, back to the browsers. It usually cannot do advanced stuff, like accessing a database in the background and sending database queries back to the browsers. CGI was invented to increase the functionality of web servers. It allows web servers to talk to CGI applications residing on the same machine as the web server itself. CGI applications do the job that the web server by itself is unable to do. For example; accessing databases in the background or conducting sophisticated searches. Web servers that use CGI, are usually able to send more information to the browsers than those that do not use CGI.

What is the significance of a 'common gateway?'

The word 'common' signifies that the convention is accepted by all web servers. The word 'gateway' represents the fact that data is obtained, as if by stepping through a 'door' or a gateway into other processes that lay on the other side. Metaphorically, a web server stands on one side of the gateway and a CGI application stand on the other side and both of them talk to each other using CGI.

What is streaming?

By the term streaming, we mean that the information is processed one character at a time in a long queue. Think of a pipe whose diameter allows only one character to pass through at any instance. The characters passing through the pipe will emerge at the other pipe end, only one character at a time, in the form of a stream.

Just like in a pipe, where water cannot go streaming in both directions; a stream in the programming world cannot be both readable and writable simultaneously. Either characters come out from the stream (which means the stream is a readable stream), or characters are placed into the stream (a writable stream) but the same stream cannot be used for both purposes by a single program.

In the case of CGI applications, the characters are in ANSI format, and so they are 8-bit characters. This point has to be noted later on, as Visual Prolog by default uses Unicode characters, which are 16-bit. There are predicates available in Visual Prolog that collect streamed ANSI characters together, so processing streams would not be a problem. Eventually, it must be pointed out, our experience of the world wide web happens because of the streaming characters arriving through the Internet pipeline into our browsers one character at a time. We just don't notice it though, because once the individual characters reach the browser, they are quickly assembled together.

What is a mime-type?

Think of mime-type as a file extension. It tells the web server the kind of files it and its clients (i.e., browsers) can handle. Most web servers determine the mime-type by the file extensions it sees. When a browser interacts with a web server, it initially informs the web server about the mime-type handling capabilities the browser has, along with the resource (file of a particular mime-type, etc.) it wants the web server to give. The web server sends over the requisite file to the browser for the browser to display. Examples are: .html, .jpg, .gif, etc. Non-recognized mime-types are either not sent to the browser by the web server, or are sent as plain ASCII or as HTML (the actual behavior depends on the web server).

As the information is sent as a stream to the browser, the web server embeds some headers at the start of the stream, containing the mime-type. See Illustration 1 for a typical example of what gets sent across from a web server to a browser. Once these files with mime-types reach the browser, it analyzes the headers, deduces the mime-type the data represents, and as per its own internal table of mime-types it can handle, it displays the data accordingly.

Illustration 1

What is a header?

A header, in the context of this tutorial, is a line of text followed by a carriage return embedded in the stream before the actual data begins. In Illustration 1, the lines you see above the data, are the headers that were passed by the web server to the browser. There is a blank line separating the headers from the actual data.

Notice the 'Content-type:...' header in the illustration. That line indicates the mime-type of the data. This particular line is specifically relevant to CGI applications, as when a web server calls on the services of a CGI application to provide data to be passed on to a browser, it is clueless on the mime-type of the data that the CGI application would be giving. Hence, by convention, the CGI application is responsible for pass this particular header before the rest of the data (separated by a blank line). Note this point carefully when we actually get around to programming CGI applications, because forgetting either the Content-type header or the blank line (between the set of headers and the actual data), or both is a common occurrence and it can lead to a lot of agony.

What is a CGI application?

A CGI application or a gateway program (see Illustration 2) is an executable file (usually an .exe file), which does not receive any information via the keyboard or the mouse, i.e., it is not an interactive application (a GUI application would be an interactive application). Such applications are known as console applications. They receive their inputs using the stdin stream and they put out their outputs into the stdout stream.
The rest of the headers are added by the web server after a CGI application does its work, and the compiled information is then passed to the browser.

Illustration 2

In case of CGI applications, once you start them, they terminate on its own without any inputs from you. You can say that they work automatically like the old DOS command line programs such as xcopy, or format, etc.

A CGI application, however, is not just any console application. It would expect the inputs as per the CGI specification and its outputs should also conform to the CGI specification.

In certain situations, a CGI application can be written as a .DLL file; but such variations are web server dependent. There are even more exotic variations possible. For example, even batch files (.bat files) can be made into simple CGI applications, provided the web server is configured to accept such files as CGI applications. Let us not get into those variations. For the purpose of this tutorial we'll assume that a CGI application is an executable, console application (.exe file), adhering to the CGI specifications.

What are stdin and stdout streams?

stdin is the name given in various programming languages, to a special stream that acts as the input stream from the underlying OS (Operating System). Programs (such as those written in Visual Prolog) can read the stdin stream, and use the characters given by that stream as their input. Similarly, stdout is a stream managed by the OS into which programs will output their characters into.

The stdout and stdin streams can be daisy chained (i.e joined end to end) from different programs. For example, a program 'A' can write into its stdout stream, and then call a program 'B', which in turn will use the stdin stream to pick up what was written by the program 'A'. This is essentially the transfer mechanism used by web servers: in the previous example, the program 'A' represents the web server and the program 'B' is the CGI application. The stdin and stdout streams are often referred together as the console.

How does the flow of information take place, using CGI?

The information flow starts from a request placed by the client (i.e., the browser) with the web server. As noted before, the client tells the web server about the various capabilities it has, and also speciies the information that it requires from the web server. See Illustration 3, which shows what gets sent by a browser (the client) to the web server.

Illustration 3

The first line contains the method field. This indicates the desired HTTP method, the resource being requested, the protocol version, etc., i.e., GET /Tests/file.html HTTP/1.0. Though /Test/file.html looks like a path, it is actually known as a URL (Uniform Resource Locator) and it represents a whole lot of things; including actual filenames and directories on the machine where the web server is located.

Other common methods are POST (for sending data from the client to the server (not as part of the URL)) and HEAD (retrieve only header information about the resource).Other fields contain information about the client. The Accept: fields tell the server the different types of data that the client can accept. These are sent as MIME Content-type messages. Thus Accept: text/plain means that the client can accept plain text files, and so on. This MIME type is important, as this is how web servers tell clients the type of data being sent. The other headers are informational: User-Agent: - a program making the request, From: - who is making the request, Referer: - gives the URL of the document making this request, and so on. The server responds by returning the desired data. It first sends server response headers, which communicate information about the state of the transaction, the type of data being sent, and any additional information. This is followed by a blank line and then the actual data being sent to the client. We saw this, in Illustration 1. Where does the CGI come into this?, - you may wonder. Well, let us re-visit the requested resource, which is in the above example /Tests/file.html, if instead of that, it was /cgi-bin/helloworld.exe. Many web servers are configured to let the partial URL /cgi-bin/ indicate that helloword.exe is a CGI program, which the WEB server must invoke.

Hence helloword.exe would be called by the web server and it would pass on some information to helloworld.exe using some environment variables and the stdout stream (which is the stdin stream for helloworld.exe). The web server will then wait till the CGI application, helloworld.exe, finishes its work.

Once helloworld.exe ceases execution, the web server would gather the data written by helloworld.exe (including the Content-type header), and send it to the waiting browser.

Phew! All that should ideally happen in the smallest period possible.

Thankfully, all web servers are designed to do its job well and so we do not have to worry about the assembly work performed by a web server after it invokes a CGI application. Let us concentrate instead on how a CGI application should read information from the web server and then pass its data back to the web server, after doing its work.

What is an environment variable?

All OS (Operating Systems) allow programmers to insert name=value relationships in what are known as environment variables, somewhere in the computer's RAM (Random Access Memory). For example; the PATH setting in DOS is an environment variable. You can examine the various environment variables that are set for DOS (or Windows XP or Windows 2000 for that matter) using the SET command. Just give SET command on a command line prompt, and the console will display all the currently available environment variables. (In the displayed list, you would see PATH also).

The best part of the concept of environment variables is that you can cook up your very own variables and assign values to them. For example; if you want to keep a password around somewhere, you could do this using the SET command at the OS command line:

SET password=xyze

Though the value is a string, you don't have to give quotes. All values are internally stored as strings, and they are retrieved by Visual Prolog programs as strings. The significance of environment variables in CGI applications is that a web server uses various environment variables also to pass information to the CGI application that it invokes. Here is an example.

We learnt that a web server places a lot of data on the stdout stream before calling a CGI application. How does the CGI application know the number of characters to read from the flowing stream? To facilitate this, the web server places the content length into an environment variable called, CONTENT_LENGTH, and using that environment variable's value the CGI application can read the right amount of characters from the stream. Let us examine some Visual Prolog code from cgi.pro (as part of the CGI pack that came with Visual Prolog) that does this:

clauses
    retrieve_POST_string() = Result :- % Get the length
        LenStr = environment::getVariable('CONTENT_LENGTH'),
        Len = toTerm(LenStr),
        %Read the stream to the desired length only
        Stream = console::getConsoleInputStream(),
        Stream:setMode(stream::ansi(ansi())),
        %The stream is in ANSI mode!
        Result = Stream:readString(Len).

Apart from CONTENT_LENGTH, most web servers set the following standard environment variables before starting a CGI application. A few additions and subtractions may be possible to these environment variables, depending on the web server that you use. The CGI application, in turn can then look for those variables and use the values that are contained therein. The reader is requested to read about these environment variables elsewhere, else this tutorial would become too lengthy. Of these, the environment variable HTTP_REFERER can be useful in implementing some simple security measures (described later in Section B of the tutorial).

PATH_INFO
PATH_TRANSLATED
REMOTE_HOST
REMOTE_ADDR
GATEWAY_INTERFACE
SCRIPT_NAME
REQUEST_METHOD
HTTP_ACCEPT
HTTP_ACCEPT_CHARSET
HTTP_ACCEPT_ENCODING
HTTP_ACCEPT_LANGUAGE
HTTP_FROM HTTP_HOST
HTTP_REFERER
HTTP_USER_AGENT
HTTP_COOKIE
QUERY_STRING
SERVER_SOFTWARE
SERVER_NAME
SERVER_PROTOCOL
SERVER_PORT
CONTENT_TYPE
CONTENT_LENGTH
USER_NAME
USER_PASSWORD
AUTH_TYPE

How do I test CGI applications?

You would need to have a web server installed, along with your CGI application in order to test it. You would also need support HTML files, which will trigger the CGI application through the web server. They will obviously vary from one application to another. You'll become aware of such support files, when you study the examples.

As indicated somewhere before, a CGI application has to be placed in the same machine, where the web server is itself running. For reasons of security, a web server is able to fetch files from only a specific set of directories (the set is known as the web-path) on the host machine. Therefore, you should place a CGI application files (CGI executable file and HTML files) into one of these directories.If that was not the case, some malicious user may try to fetch files that should not have been retrievable.

Similarly, a web server can only invoke a CGI application if it is placed in the right directory. You cannot place the CGI application anywhere on the web-path (well, a few web server do allow that too, but not all). Now, all web servers have a configuration method (a separate configuration file, or an .ini file or the windows registry, etc.) to configure its various settings. One of the crucial settings is the location of the CGI applications. When you zero in on a web server for your CGI application, the first thing you should study is how that particular web server configures its CGI application directories.

Which web server to use?

As indicated before, web servers are not complicated pieces of software. There are many web servers available free of cost. A large list can be seen at: http://www.serverwatch.com/stypes/index.php/d2Vi . That list includes both commercial software as well as free ones. The complexity of the web server is usually the configuration of the web-path and CGI application directory/directories of the web server.

The TCP-IP protocol should be installed in the computer where the web server is to be setup (It usually would be, if you are able to browse the Internet from that computer) Once you have successfully installed a web server, you should be able to get data from the web server by supplying either the IP address or the domain name of the machine where the web server is installed. Please note that the CGI programs should work fine with the most of web servers including the IIS5 web server which is usually included with Windows 2000 by Microsoft.

Where to place the CGI application?

Ensure that the CGI application is placed correctly in such a web server's appropriate script directory. This will vary from a web server to a web server. In this tutorial, we will assume that you have placed it in such a directory that it can be invoked using a browser on the server machine with the following URL: http://localhost/cgi-bin/<APPNAME>.

For example, if you have a CGI application called example1.exe, then it should be accessible with the following URL, on the server machine: http://localhost/cgi-bin/example1.exe.

Can you show me a very simple CGI application created using Visual Prolog?

Yes, indeed! You should unzip the file example1.zip that is included into cgitutorial.zip into a convenient place and examine the Visual Prolog files therein. But, you can create such an application from scratch too by following the instructions given below.

Start a new Visual Prolog project. Note that the UI Strategy is Console:

The main Project tree would get displayed. It will include all the necessary files that the IDE has intelligently decided you may require.

Now, click on the main Build menu, and in that menu, select Build or press Ctrl-Shift-B. You'll notice something magical, as the build process continues. As the project is being built, the IDE will pull in all the relevant PFC (Prolog Foundation Classes) files that were not visible earlier in the Project Tree. After the build process is over, the Project Tree will now look like this:

Congratulations! You have finished 95% of the job. Now you would be having a console application in the EXE folder of your project, which currently does nothing. We need now to add the relevant code to transform the application into a CGI application

As explained in the earlier sections of this tutorial, a CGI application expects some input from the stdin stream (Along with some environment variables), then it does its work and sends out the output through the stdout stream. As this is a very simple CGI application (our very first one) we'll ignore all input arriving into our CGI application from the stdin stream, (which the web server may be sending to the CGI application). Instead, we'll make our little application send some output.

Double click on the file main.pro in the Project Tree, and go towards the end of the code:

You'll notice that the main goal of the program invokes a predicate called run from the PFC module mainEXE. That module in turns invokes the predicate called run() from main.pro, so let us have a look at that:

clauses
    run():-
console::init(),
succeed().  % place your own code here

We'll modify that code to make it look like this:

clauses
    run():-
        console::init(), %Line 1
        cgi::init(stream::ansi(core::ansi)), %Line 2
        stdIO::write("Content-type:   text/html\n"), %Line 3
        stdIO::write("\n"), %Line 4
        stdIO::write("<html><body><h1>Hello World!</h1></body></html>"). %Line 5

Let us now go through the code, line by line:

• Line 1: is a Visual Prolog requirement. All work using the console, should first start with console::init() .

• Line 2: captures the stdout stream object to CGI output. As explained in the paragraph of this tutorial, the CGI application is expected to send its output to the stdout.

• Line 3: Now the CGI application is prepared to send the first line back to the calling web server. As explained in the paragraph of this tutorial, the data has to be preceded by some headers. The header which is mandatory is the one that sets the mime-type of the data. Here, it states that the Content-type of the data is of the mime-type text/html. Notice that the line ends in a carriage return (\n).

• Line 4: To separate the headers from the rest of the data a blank line is needed. It is mandatory as per the CGI specification. So the CGI application now sends a blank line (\n) there. Forgetting to do this is very common in CGI applications (Read the paragraph of this tutorial for more), and it can lead to the malfunctioning of the application.

• Line 5: In the end the actual data is written out to the stdout. In this example, it is simply creating an HTML file and sending that. The HTML file contains just two words : Hello World!.

How to test Example1.exe?

Place it in the script directory of your web server, as explained earlier. Then access the following URL on the same machine as the web server http://localhost/cgi-bin/example1.exe using a suitable browser.

What is the functionality of such a primitive application?

The Hello World CGI application (Example1) is not actually as primitive as it seems. It is simple alright, but it can serve as the seed for much sophisticated stuff. Take the last line (Line 6 explained in Paragraph of this tutorial). There the complete HTML is created and then dispatched over the stdout stream to the web server, who in turn, sends it to the requesting browser. If you introduce a database access at that point, and add some more predicates for conditional processing of the data from the background databases, you can easily build a content management system

What about inputs?

As indicated earlier, we have ignored whatever inputs the web server may have given to our little application. If the CGI application needed some inputs, then the stdin stream would have to be read, and processed for information. Also, the environment variables set at the time of the CGI application operation would also have to be examined. This can be a little complex. Luckily for us, we do not have to do much programming in Visual Prolog when developing CGI applications. For example, to know what data the CGI application receives from the web server, all we have to do is to use the cgi::getString() predicate. (We have to include the cgi.pack from the PFC into our project). And if we need the data in a more convenient form, then we can use the cgi::getParamList() predicate. The individual pieces of data are assembled by cgi::getParamList() into name, value pairs similar to environment variables.